Agreed. I'm at -7 and I didn't do anything to deserve this. It's quite rude and unfriendly. But it clearly makes some people feel good to have the power to do that, certainly because they can't do it IRL. Poor sapiens.
I understand how this can be frustrating, but it is not true that you didn't do anything to deserve this. You published several comments with high noise-to-content ration (and yes, complaining about downvotes is also noise). You got downvotes for that. Take them, move on and do better next time. It's just bits on some server, and they mean nothing except showing you how to improve your participation on the site.