The human neocortex has 20B neurons, averaging 10K connections each, which is about 200T connections total. This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category.
Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general).
All I'm trying to say is I find it crazy how dang big these models are getting.
>> This model is only a few orders of magnitude away from that, and it's already performing really well in its narrow category.
A few orders of magnitude and an entire category away. Artificial "neurons" only have the name "neuron" in common with biological neurons. Consequently you can stack as many layers of artificial neurons on top of each other as you may want and you won't get anywhere near the abilities of the simplest systems of biological neurons.
For example, spiders have ~100 thousand neurons and there's no artificial neural networks that could shake a stick at a spider's cognitive abilities. Which are downright scary, btw.
Estimated number of neurons of spiders from wikipedia:
Spiking ones need to solve differential equations and our current hardware designs are too discrete for it to be an efficient strategy (barring an algorithmic breakthrough).
> This model is only a few orders of magnitude away from that
I think you can add several orders of magnitude to that since nerve cells are more like microcontrollers (with memory, adaptation etc.) than simple nodes. I remember a scientific article that made a big impression on me: when a dragon fly sees a prey, only 8 neurons (connected to the eyes and the wings) are responsible for keeping it oriented toward the target.
For all we know these microcontrollers may be that complicated only because they try to emulate discrete logic, and you actually need hundreds of them to make a single unit reliable enough for that purpose.
Equating model 'parameters' to interneuron connections in naïve at best (and a horrible measure in general).
All I'm trying to say is I find it crazy how dang big these models are getting.