The threshold is 500, as far as I know. Whenever I reach that, I delete my account and create a new one. I'm probably one of those people who downvotes others that I don't agree with, and I don't want that power. Especially since, imho, 50% of the comments that are downvoted here don't deserve it.