The models are a lot of fun to play with, but yeah, every time I've tried to use them for something "serious" they nearly always invent stuff (and are so convincing in how they write about it!).
Most recently I've been interested in what's happened with the 4-color theorem since the 1976 computer-assisted proof, and decided to use GPTChat instead of google+wikipedia. GPTChat had me convinced and excited that, apparently the computer-assisted part of the proof has been getting steadily smaller and smaller over the years and decades, and we're getting close to a proof that might not need computer assistance at all. It wrote really convincingly about it! And then I went and looked for the papers it had talked about. They didn't exist, and their authors either didn't exist, or worked in completely unrelated fields.
Before the inevitable idiots come in to say hurr durr but have you tried ChatGPT 4… yes I paid for it, and it is just as prone to hallucinations of factual information. It loves to make up new names for peoples initials.
While it is still prone to hallucinations, with GPT4 I've had it tell me "X is not a real project, but assuming it exists and is designed to do Y [based on the name], I can provide you with a general guide on how to use it." I never had earlier version to do that. So it does appear to be better, though it's certainly still an issue.
Most recently I've been interested in what's happened with the 4-color theorem since the 1976 computer-assisted proof, and decided to use GPTChat instead of google+wikipedia. GPTChat had me convinced and excited that, apparently the computer-assisted part of the proof has been getting steadily smaller and smaller over the years and decades, and we're getting close to a proof that might not need computer assistance at all. It wrote really convincingly about it! And then I went and looked for the papers it had talked about. They didn't exist, and their authors either didn't exist, or worked in completely unrelated fields.