Kinda, but it could harm your future prospects though if you’re unaware you’re getting badmouthed in your absence and word gets round. One degree of separation is all it would take before people who’ve never met or worked with you have been persuaded you’re a problem.

... still my advice is that life is too short to care about such things. In my experience people (guys too) that indulge a bit too much in rumors are massively insecure folks with various other issues, and it shows.

Have confidence in your skills and the rest are details, especially if we talk about IT where job mobility is generally way above average, you really don't have to kiss assess or be worried what others think/say about you to be successful.