It's because there are two types of voters: those that vote because they agree or disagree with a sentiment and those that vote because the comment was useful and added to the conversation. We want more of the second group voting; the first group's votes are useless and counterproductive.

Your first comment completely deserves a downvote by the second group, and your second comment is intresting but off-topic; I generally vote those up if I vote on it at all.