It's important to recognize that the cortex is a relatively late addition to the structure of the brain. Brains have been learning about the world and how to behave appropriately long before some mammals developed such a large region here. It's much more likely that the core functionality to be an autonomous agent in the world is encoded in the limbic system (e.g. basal ganglia, hypothalamus, mid brain and brain stem regions).
Certainly those regions are doing something in humans; and certainly those regions had those functions in species without a cortex. But the implication that these regions' presence in the human brain implies they're the ultimate arbiter of the same functionality there, isn't guaranteed to be true.
Brains really do use "subsumption" (in the https://en.wikipedia.org/wiki/Subsumption_architecture sense) to accomplish various features; there are motor signals that some earlier-evolved part of the brain would be emitting if they were the only thing "online" in the brain, that are actively suppressed or "overridden" by another, later-evolved part of the brain, often a part that only "comes online" later in brain development. (Thus "early instincts" that disappear during development, like the infant diving reflex.) Often, these subsumed neural processes reappear when the region suppressing them is damaged, as in the normally-suppressed human lordosis reflex, or the normally-suppressed-after-infancy suckling reflex in most mammals.
There's no evidence that I know of that the limbic system and the cortex are in this sort of relationship; I'm just saying that this kind of relationship isn't unprecedented as a thing human brains do.
If such a relationship were to exist between the cortex and the limbic system, then both regions could be said to be "in charge" of cognition; sort of like a television tuner and a VCR can both be in charge of the image being displayed. One state machine (the TV; the cortex), passing input through to another state machine (the VCR; the limbic system) only in some subset of its states (the right "channel"; the right arousal state.)
> There's no evidence that I know of that the limbic system and the cortex are in this sort of relationship
It's true that in mammals (possessing a neocortex) the cortex takes over various processes traditionally performed by the limbic system. Complete removal of the cortex in rats demonstrates that behaviour can revert to entirely limbic control: https://www.ncbi.nlm.nih.gov/pubmed/564358
While the cortex can definitely be / become "in charge" of complex intelligent behaviours, it may still need the brain's phylogenically older machinery to bootstrap it.
"It does this by decomposing the complete behavior into sub-behaviors. These sub-behaviors are organized into a hierarchy of layers. Each layer implements a particular level of behavioral competence, and higher levels are able to subsume lower levels (= integrate/combine lower levels to a more comprehensive whole) in order to create viable behavior. For example, a robot's lowest layer could be "avoid an object". The second layer would be "wander around", which runs beneath the third layer "explore the world". Because a robot must have the ability to "avoid objects" in order to "wander around" effectively, the subsumption architecture creates a system in which the higher layers utilize the lower-level competencies. "
One can imagine how backpropagation style training on "explore the world" or other high-level scenarios would result in formation of "avoid objects" kernels at the lower levels similar to how image recognition deep nets produce Gabor like kernels at the lower levels. We do have some theorems on optimality of such deep networks for image recognition and i wouldn't be surprised if similar optimality was present for behavioral deep networks. Also, it seems that like in case of image deep nets, the deep architecture for behavior naturally allows for transfer learning by reusing the lower layers too - like one would reuse the low/mid layers of image deep net, one would naturally reuse the lower "avoid objects" layers (and may be some more complex aggregate behaviors from "mid-levels") for other tasks.
The architecture of the subsumption system is fairly well mapped: from basic optical edge detectors in the banks of the calcarine sulcus to high levels of sophistication as you move forward (object recognition, numeracy, vocabulary in the parietal, then motor function, then judgement and grammar in the frontal and prefrontal cortical areas). The additional dynamic of taking over or controlling traditionally limbic behavior (emotion) is an additional process which I would characterize as a blend of subsumption and adaption: the cortex aquires data and abstracts it, but also, as an adaptive response.
Exactly. I think when people talk about intelligent machines today what they are most after is autonomous, robust, common sense orientation and agency. It's not the ability to do algebra or reason logically, because we can actually already built systems that are quite good at that.
What we really want is a robot or a car that can orient itself in the abscence of structured data and doesn't glitch out into the wall once it loses its objective. And even primitive animals are very good at that.
