This might be true. Your logic applies to all such cases though. Right to vote suffers from exactly the same drawbacks as right for speech. And authoritarianism has the same tempting benefits as censorship -- you can stop people with bad ideas from messing things up. And, just like with censorship, one day you might end up on the wrong end of the system, and not necessarily because your ideas are bad.