Well, when you downvote, you're really saying "The community should have less of this", not "I want less of this". Karma is global, not different depending on which user looks at it.

Therefore, if the community has deemed a comment as stupid (karma < 1), any future readers will read the comment through tinted glasses. But wait a minute, the community really hasn't said that. It was only one or two users that did.

Let's reexamine the purpose of karma on a comment. The purpose is not simply to punish the poster of a comment if it gets downvoted. It's to move the noise down to the bottom of the thread, where it can be ignored.

So I propose an alternative: Let silly comments stay at the bottom with one karma point, and encourage users to upvote more interesting comments higher. This way, users won't be biased against any comments (They won't know if a comment has been downvoted because it might simply be that no one has upvoted it) and the noise can still stay at the bottom.

The problem is that each vote change the number of points, so if you happen to be the first down-voter, it will become zero. But if another user up-voted the same item just before you, then your vote change nothing.

I seems that reddit solved this problem by not showing the number of points for new messages.

There could be a "new" status, marked in a special way, which explain why you vote it not showing. After enough users voted, and using the editor vote to change the weight of other votes, move the item to voted status, and show its point count.