I think it's worthwhile to differentiate between a critical mass fission/fusion bomb and a dirty bomb. A dirty bomb can be made from anything radioactive coupled with a conventional explosive, since the goal is just to spread the radioactive material as widely as possible. Fission/fusion bombs (what we typically would consider as "atomic bombs") are, as you said, way harder to pull off. The general public definitely likes to conflate the two.