You'll always run into that problem if you care. Most of the time this results in having a clique against you who'll make your life hard, for example with increased bureaucracy.

If the manager is on your side, it's possible but not fun.