You are right that there are many articles in the open describing nuclear bombs. Still, to actally make them,is another big leap.
Now imagine the AI gets better and better within the next 5 years and is able to provide and explain, in ELI5-style, how to step by step (illegaly) obtain the equipment and materials to do so without getting caught, and provide a detailed recipe. I do not think this is such a stretch. Hence this so called oh-my-gosh limitations nonsense is not so far-fetched.
Now imagine the AI gets better and better within the next 5 years and is able to provide and explain, in ELI5-style, how to step by step ... create a system to catch the people trying to do the above.
That you think that there's like a handful of clever tricks that an AI can bestow upon some child and ta-da they can build a nuclear bomb in their basement is hilarious.
What an AI would almost certainly tell you is that building an atomic bomb is no joke, even if you have access to a nuclear reactor, have the budget of a nation-state, and can direct an entire team of trained nuclear physicists to work on the project for years.
Next thing you'll be concerned about toddlers launching lasers into orbit and dominating the Earth from space.
5 years from now, not only AI will be more advanced. Also techniques and machinery to make things will be more advanced. Just think about other existing technologic advancements and how absurdly 'ta-da' they would have sounded not too long ago.
How is that a good reason for GPT4 not being able to write the word 'fuck'? You might handwave the patronising attitude of OpenAI strategy, but with many of ust they did lost most of their good faith by trying to make their model 'safe' to a horny 10-year-old.
It is a massive stretch given how well the materials are policed or how much effort is required to make them. There is no reason to assume that there is some magic shortcut that AI will discover.
Now imagine the AI gets better and better within the next 5 years and is able to provide and explain, in ELI5-style, how to step by step (illegaly) obtain the equipment and materials to do so without getting caught, and provide a detailed recipe. I do not think this is such a stretch. Hence this so called oh-my-gosh limitations nonsense is not so far-fetched.