People do a lot of things for short term pleasure that risk heavy long term consequences. Societies don't have the right to decide for others what their choices should be.

Maybe outlaw ice cream next? Or how about marijuana smoking? (Oops, already tried that.) How about loud music (long term hearing damage)? How about motorcycles?

> Societies don't have the right to decide for others what their choices should be.

If an AGI starts inventing new and wonderful ways in which we can destroy ourselves, and we are taken in by it, we will have to restrict that, it's non optional.

You can make the argument that, say, current hard drugs should be legal - but I don't think there's a way to defend the position that any possible future 'thing' should always be legal/permitted regardless of negative effect.

An awful lot of harm has been done to people via one group who are sure they know what is best for others, and thereby are justified in forcing it upon them.

In fact, likely much more harm than those with evil intent.