That's a natural consequence of the human condition, coupled with a libertarian outlook when creating moderation tools. A downvote can never be presumed to mean "this is objectively wrong", always "this is subjectively objectionable"

A good exercise in knowing thyself, examining your own prejudices and improving the world around you because of that critical introspection. Unfortunately, this process must be repeated near-constantly to bring about perfection, and few people have the time for that (or are willing to make the time for that)