I think this is a fallacy of thinking that we can easily explain everything and look at it through an anthropomorphic lens.
If you take away the humans, both the input and the output have no meaning. What the "AI" does is just a computation. If someone claimed that the OS on your laptop is AI you would say they're nuts - yet through the multiple layers of abstraction and the hardware/software synergy an external observer who does not understand how hardware and software work could reasonably make the assumption that it's alive and thinking.
Things can be really complex and fully explainable at the same time. A program could have millions of variables and you could zoom in on a variable and explain its purpose exactly.
In a neural net, the creators of the neural net in general don't know what it means for the weight at some node being 0.46557, or how the system would behave if you changed it to 0.5.
CNNs tend to use the first few layers to detect surface features like edges, and later layers to detect more abstract features. This was an empirical discovery! The creators of the CNNs were like holy shit, these layers detect edges!
Anyway I think there's a substantial difference between building really complex systems (that yes may appear as black boxes to outsiders) and systems where the designers themselves generally don't know how the thing works.
Yes, i agree with you. I'm just saying that once you're past a certain complexity threshold and you cannot abstract and encapsulate lower level components to make higher level ones and they all make sense at their level and you can zoom in and out it's hard to reason about any system.
If you take away the humans, both the input and the output have no meaning. What the "AI" does is just a computation. If someone claimed that the OS on your laptop is AI you would say they're nuts - yet through the multiple layers of abstraction and the hardware/software synergy an external observer who does not understand how hardware and software work could reasonably make the assumption that it's alive and thinking.