The fact that something causes me anguish isn't the only factor in whether other people should be allowed to do that thing.
E.g. maybe I've had traumatic experiences around being called stupid, and whenever I perceive others as thinking I'm dumb, I remove those experiences. Should it be illegal for people to call me an idiot in such cases?