> in my experience LLMs hallucinate so much on stuffs I know about that I instinctively challenge most of their assumptions and outputs

In my experience most people don’t do that. Therein lies the problem.