I think a lot of the controversy dates back to the anti-war and civil rights movements. Psychedelics were outlawed to tackle the first, opioids to address the latter. Now there were "legitimate" charges to ring them up for instead of civil disobedience.

Result: fewer challenges to the status quo when fundamental change at near-revolutionary levels seemed a real risk for those in power. Turns out society's elite were a bunch of squares.