Everyone is an asshole in someone's mind. You'll never please all people, and when you don't, they'll think you're an asshole. That's life. Don't let what they think bother you.
If one person calls you a jackass, ignore him. If two people call you a jackass, think about it. If three people call you a jackass, buy yourself a saddle.
Well, I'd rather 50 people hate me and 50 people love me than 100 people have a neutral opinion towards me. Those aren't necessarily the only options, but a lot of personality traits/beliefs tend to be highly polarizing.
For example, I hate it when people spend a long time deciding where to eat. It drives me crazy. So if it goes on for more than 5 minutes, I'll say "I'm going to (insert restaurant here.) Anyone who wants to join me should come" and leave. Some people love this, some think I'm an arrogant prick, and most are just happy they don't have to wait 45 minutes to eat.
So really, you should ask why people think you're a jackass. I know some people hate my tendency to unilaterally make lunch decisions, but I have no intention of changing it.
Though, one person out of ten is less significant than 100/1000. We should work out the probabilities of someone telling you you're a jackass given that you are/are not one, and update the proverb to use likelihood ratios.