He did not say what kind of research strategies or techniques might take its place. In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.
> In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size.
I read the two papers (gpt 4 tech report, and sparks of agi) and in my opinion they don't support this conclusion. They don't even say how big GPT-4 is, because "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."
> Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.
OK so his argument is like "the giant robots won't be powerful, but we won't show how big our robots are, and besides, there are physical limits to how giant of a robot we can build and how quickly we can build it." I feel like this argument is sus.
OpenAI has likely run into a wall (or is about to) for model size given it's funding amount/structure[1] - unlike its competition who actually own data centers and have lower marginsl costs. It's just like when peak-iPad Apple claimed that a "post-PC" age was upon us.
1. What terms could Microsoft wring out of OpenAI for another funding round?
He did not say what kind of research strategies or techniques might take its place. In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.