Too many hallucinations because changes in fast changing libraries. It makes up functions that were removed and libraries that have been deprecated. With robust stuff that doesn’t happen. It is frustrating. Even copilot regularly includes some react lib that doesn’t work anymore for the past 3 years because it depended on old stuff that is full of security issues, bugs or doesn’t work on a new node etc etc. So then I spend more time on finding what is the newest (greatest … cough) thing to replace it than I would have done just searching google.
It’s funny because a big draw of React is the sizable user base and library count, but that ends up being a double edged sword when using LLMs for codegen.
Well, if they are not up to date I guess with the latest; it changes so fast and there are so many examples of all different flavours of how to do something... I guess they will find a mechanism which will make newer info more relevant and older info more forgotten in some cases and also a way to draw in the latest all the time. My problem is more the not built/designed etc here issue; when you have a react WidgetBlah library and someone else also wrote one, there are reasons you rolled your own (often also not but ok), but why didn’t you keep the api the same. People talk about features here as a reason for breakage but when I compare a breaking feature change (for instance, a react component sig changed from one version to another) then I hardly ever see a reason why that was not made backward compatible. I have complained about this in many GitHub issues and the answer is usually ‘this way is better, just refactor it: it’s easy’. Sure it’s easy but it’s work and this happens a little too often. Sometimes I would like Java standard bodies and voting about api changes and this is one of those.
RAG and large context sizes mitigate this well enough for me. Ingest the library's docs (and maybe a sizable chunk of your codebase) and use that to get better LLM output that isn't out of date or hallucinated.
