I wouldn't worry much. I have a few "fans" that follow my comments regularly and mass down-vote me. I also realized lately that my own upvote/downvote does not get accounted for.