You could set up a karma threshold to vote, or perhaps allot an increasing number of votes per day based on karma. It doesn't depart greatly from the present karma thresholds and would limit the drowning of commenter votes by reader votes.