If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do!
Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware.
Machine learning is actually premised on being “simple” to implement. The more priors you hardcode with clever algorithms, the closer you get to what we already have. The point is to automate the process of learning. We do this now with relatively simple loss functions and models containing relatively simple parameters. The main stipulation is that they are all defined to be continuous so that you can use the chain rule from calculus to calculate the error with respect to every parameter without taking so long that it would never finish.
I agree that your suggested approach of applying cleverness to what we have now will probably produce better results. But that’s not going to stop better architectures, hardware and even entire regimes from being developed until we approach AGI.
My suspicion is that there’s still a few breakthroughs waiting to be made. I also suspect that sufficiently advanced models will make such breakthroughs easier to discover.
I have repeatedly argued against this notion of „just predicting the next word“. No. It‘s completing a conversation. It‘s true that it is doing this word by word, but it‘s kind of like saying a CNN is just predicting a label. Sure, but how? It‘s not doing it directly. It‘s doing it by recovering a lot of structure and in the end boiling that down to a label. Likewise a network trained to predict the next word may very well have worked out the whole sentence (implicitly, not as a text) in order to generate the next word.
I actually have high hopes for the hybrid architecture Ben Goertzel has been working on at OpenCog. I think the LLMs are soon going to hit a S curve w/o introduction of additional scientific knowledge like physics and notion of energy (wrt AGI development, they'll still be good for tonnes of other jobs displacing things).
FWIW we had thin clients in computer labs in middle school / high school 15 years ago (and still today these are common in enterprise environments, e.g. Citrix).
Biggest issue is network latency which is limited by the speed of light, so I imagine if computers in 10 years require resources not available locally it would likely be a local/cloud hybrid model.
Personally, I'm less worried about AI than I am about what people using these models can do to others. Misinformation/disinformation, more believable scams, stuff like that.
Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware.
It is still going to get wild