You want something that requires an engineering breakthrough.
Models don't have memory, and they don't have understanding or intelligence beyond what they learned in training.
You give them some text (as context), and they predict what should come after (as the answer).
They’re trained to predict over some context size, and what makes them good is that they learn to model relationships across that context in many dimensions. A word in the middle can affect the probability of a word at the end.
If you insanely scale the training and inference to handle massive contexts, which is currently far too expensive, you run into another problem: the model can’t reliably tell which parts of that huge context are relevant. Irrelevant or weakly related tokens dilute the signal and bias it in the wrong direction, the distribution flatten or just ends up in the wrong place.
That's why you have to make sure you give it relevant well attended context, aka, context engineering.
It won't be able to look at a 100kloc code base and figure out what's relevant to the problem at hand, and what is irrelevant. You have to do that part yourself.
Or what some people do, is you can try to automate that part a little as well by using another model to go research and build that context. That's where people say the research->plan->build loop. And it's best to keep to small tasks, otherwise the context needing for a big task will be too big.
> You want something that requires an engineering breakthrough.
Basically, yes. I know the way LLMs currently work wouldn't be able to provide what i want, but what i want is a different way that does :-P (perhaps not even using LLMs).
Models don't have memory, and they don't have understanding or intelligence beyond what they learned in training.
You give them some text (as context), and they predict what should come after (as the answer).
They’re trained to predict over some context size, and what makes them good is that they learn to model relationships across that context in many dimensions. A word in the middle can affect the probability of a word at the end.
If you insanely scale the training and inference to handle massive contexts, which is currently far too expensive, you run into another problem: the model can’t reliably tell which parts of that huge context are relevant. Irrelevant or weakly related tokens dilute the signal and bias it in the wrong direction, the distribution flatten or just ends up in the wrong place.
That's why you have to make sure you give it relevant well attended context, aka, context engineering.
It won't be able to look at a 100kloc code base and figure out what's relevant to the problem at hand, and what is irrelevant. You have to do that part yourself.
Or what some people do, is you can try to automate that part a little as well by using another model to go research and build that context. That's where people say the research->plan->build loop. And it's best to keep to small tasks, otherwise the context needing for a big task will be too big.