a baby is still predisposed to learning language (amongst many other things essential for human living/survival) thanks to the brain and evolution. No human is really starting from scratch in any meaningful way.
If you look at Chess, Poker or writing Python, I am not sure that natural evolution is giving us a huge head start.
And still, human experts in those fields don’t need as much data, even with our slow brains, the convergence rate is astounding, compared to machine learning.
I’d say that an understanding of causality helps people learn chess. Maths, from counting through algebra helps people learn to program. I’d imagine it would be hard to understand the concept of a loop if you couldn’t count.
> If you look at Chess, Poker or writing Python, I am not sure that natural evolution is giving us a huge head start.
The point you're missing is that those games have been designed by humans, for humans, so even if the natural selection didn't give us any advantage in playing chess per se, it conditioned our brain in a way that made us invent chess in the first place.
That being said, the original argument of comparing NN training data and natural selection is stupid anyway.
I think you're missing the forest for the trees here. First of all it's not well understood how infants are so able to learn languages, and the extent to which language ability is innate is fairly controversial in linguistics.
Leaving the details aside, the fact that a human is not starting from scratch is not in dispute. But the whole point of the discussion it seems to me is the question of exactly how humans are not starting from scratch, i.e why do we learn so much faster, and how could we apply the answer to current techniques in machine learning?
Those are still interesting questions whether or not humans and randomly wired neural nets are both starting from scratch.
A pretty obvious difference is that these models are still nowhere near as large or complex as a human brain. This network has 15 billion parameters, whereas a human brain is estimated to have 60 trillion neuronal connections. Additionally each neuron, of which a human brain has around 90 billion, can fulfill many more roles than a "neuron" in a language model.
Apples to oranges, but there's a pretty obvious complexity gap.
The neurons in Wernicke's area however is a very small subset of this, so since these models aren't doing anything related to taste or smell, etc that number isn't as relevant as you may think it is. The number of neurons more dedicated towards proprioception for example is quite vast, and often almost completely undiscussed by the AI community.
So you're not making quite the argument that you think you are; although the general idea that there's still a difference is obviously true (birds v planes, yada yada).
Yep. Also of note is the fact that human learning is fundamentally a more flexible process in that it can lay down new neuronal connections and in fact new neurons too.
I'm sure there are (evolutionary?) NN models that try to do things like this but I have no idea how successful they've been.
>First of all it's not well understood how infants are so able to learn languages
It's not well understood sure but the brain is evidently playing a crucial process. Children learn to speak languages at about the same time with the same milestones occurring at roughly the same ages. Not to mention the fact that despite wildly different cultures and situations (some cultures don't attempt correct their children ever, some cultures don't speak to babies), children learn language just fine. Controversy on exactly how much aside, we're obviously predisposed to it.
>But the whole point of the discussion it seems to me is the question of exactly how humans are not starting from scratch, i.e why do we learn so much faster, and how could we apply the answer to current techniques in machine learning?
The closest biological equivalent to a parameter in an ann is a synapse. Well humans have about 100 trillion synapses. We already know that the higher the parameter count, the lower the training data required. a 50 billion parameter model will far outperform a 5 billion one trained on the same data. and a 500b one would far outperform that 50 billion one.
Economics limits how far we can go and i'm not making any declarative statements but who's to say that's not the issue ?
ann and whatever the brain does diverged in details a long time ago. It's cool to speculate and all but any special insight on the brain would have little implications on the future of deep learning. That's just not what drives architectural advances.
we could have expert level machines in a couple years but any approach trying to copy the brain is decades if not centuries away. That's how little we understand. and how little impact that actually has on the DL of today.
Current LLMS are actually nowhere near the scale of the human brain, either in parameters/neurons or training data (all the text we've ever trained an LLM on would be dwarfed by all the data humans perceive). as well as not having the headstart the human brain has. It's kind of a bogus comparison when you think about. You could easily make the case that LLMs are far more effective.
Is my understanding of your argument that the near 24/7 auditory/visualize constant input + the brain having way more neurons helps or converge faster? I can buy that. The challenge of course is someone like Hellen Keller who had very little input in terms of quantity and yet still managed to develop into an intelligent adult once we figured out how to communicate with someone like that.
The weak spot of my argument is that it took me 20 years of on and off training, maybe 4-12 hours per day most days to get to this state. By comparison AI gets to maybe my experience level after a few years or so in months. So maybe it doesn’t actually take that much time comparatively (despite having a much lower ceiling).
The part that I’m not quite sold on though is the comparison on number of neurons. We don’t actually have a good handle on how many neurons are equivalent and a non-trivial amount of a brain’s neural net is responsible for real time signals processing of high fidelity audio and video, propiecption, motor controls etc, running your body, filtering and converting inputs into long term storage + combining it all with higher order executive functions and thought that can override a good chunk of it. It doesn’t feel like the strongest argument to make to say all that complexity is needed to create human-level intelligence in terms of comparing neurons (there may be reasons those are things are needed, but creating an LLM with the same number of neurons probably won’t work).
The compelling part for me is to continue the analogy of the brain which motivated this line of AI research. We know that the brain has all sorts of different structures and they map pretty closely to different functions and it’s not just one giant language center. Wouldn’t it make sense that we’d need different kinds of AI models to build a fully functional AI? Not least of which because specialization can be computationally more efficient (eg various computational imagery tasks are doing extraordinary things and they’re not just throwing large and larger LLMs at the problem)
