LLMs can integrate new information but they only remember what you told them up to the context length. If they could remember longer then you would have agents with a lot of new knowledge.
I guess what I am getting at is: When I ask ChatGPT 4 Turbo to "search for the article by Eric Berger from Sep 10th 2023, and summarize it," it does, and responds including that new information because it is now in the context.
The next step would seem to be to integrate those search results into its training data - while somehow filtering bunk data which is the tough part - and when the next user asks that question it already knows it without searching.
Is this the same thing as saying "infinite context length?" I didn't think so. Is it?
What I mean is if context length wasn't a limit, you could have your own AI agent that remembers that article forever, along with everything else you've ever discussed with it.
If it generated text on its own, without human interaction, and developed its "personality", then you could say it's more like what we consider an intelligent being.
Context length is limited by the length of articles in the training data. You couldn’t use infinite context lengths unless the LLM had been trained on infinity long articles.
Context lengths aren’t arbitrary limits - they reflect the ability (and limits) of the model to actually reason about lengthy topics. If you want longer context lengths you’ll need more training data, larger models, and a lot more money.
