This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests, and is mainly just a dude saying "well, certainly this seems like it would be outside the training set and the LLM seems to handle it OK so it must be reasoning". Meanwhile, we have no way of inspecting the training set, and even if we did the data and model are so massive it would be impossible to tell whether it was doing simple substitution or whatever.
Ultimately e.g. Brooks must be right that the model has no connection to the world, because to the model there is no world, only text. If the training text includes things that are untrue in the world, the model cannot access the world to make a final determination that its internal statistics are faulty. If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.
> If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.
You could say the same for human perception, though. We don't actually "see" what our brain thinks we see, our brain fills in the image. Nonetheless, it doesn't stop us from having and using our slightly-inaccurate world-model based on this info. Also, we can often infer our faulty perceptions by comparative methods like "banana for scale" or looking for conflicts and contradictions.
I think there is a pretty sizeable difference between "human perception of the world is necessarily mediated by sense organs" and "all human textual output exhaustively covers the world and can be used to comprehensively describe and navigate it". Unless we are solipsists, we tend to agree that the world maintains a presence whether we are sensing it or not, and that even our senses are merely a limited reflection of what the world is and not the ground truth; that when our ideas or senses don't match the world, the world wins. For an LLM however there is nothing outside of text.
First, GPT-4 was trained on images and text, not just text. The images improved its text predictions. Because it has a world model, and they helped populate it. It outputs only text. But just like a Unix program writing text to stdout, nothing about the fact that your output format is constrained restricts the kind of computation you can perform in service of that output.
Input: text and images
Computation: ?
Output: text
I think you're suggesting/asserting that the computation step -- the hidden layers -- must also be focused on text. But there's no such constraint in reality.
I don't think it's such a stretch to see the billions of written words given to GPT-4 as essentially a new kind of sense organ. They make it capable of rejecting the untrue claims made in its training set, because (a) the untrue claims are massively overwhelmed in number by the true claims, and (b) the true claims usually come with links to other knowledge that reinforces them.
> This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests
> Ultimately e.g. Brooks must be right that the model has no connection to the world
Isn't that a huge double-standard? If you want to say the AI has a world model, no matter how many supporting examples of informal experiences you bring up, they don't count because you don't have formal justification. But if you want to say the AI doesn't have a world model, you don't need formal justification or even any supporting data or falsifiable predictions, you just say "it couldn't possibly be the" case, and that's enough.
(Also, the whole OthelloGPT thing seems as close to formal evidence for an internal world model as you can get.)
That's why I really hated the Sebastian Bubeck presentation. It was a guy showing us all this amazing stuff in this really overenthusiastic way, which we all just had to trust was amazing because we're not allowed to see what's in the pie or have access to the same model.
Ultimately e.g. Brooks must be right that the model has no connection to the world, because to the model there is no world, only text. If the training text includes things that are untrue in the world, the model cannot access the world to make a final determination that its internal statistics are faulty. If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.