That's just a natural consequence of building a censorship-free platform; many of your early adopters are going to be people who find freedom from censorship to be particularly valuable to them.

Tor, Monero, etc all suffer from basically the same problem. Ultimately, if you value freedom from censorship, you shouldn't let the association with others who share that value (though perhaps for different, less noble reasons than you) bother you. Otherwise no platform that has that property will ever take off.