I don't understand why you would draw this conclusion. The deep search you describe is an algorithm that humans can understand perfectly fine. Humans just can't solve it in their heads and need to let a computer handle the number crunching. Just like a scientist may understand the differential equations to describe a system perfectly fine, but require a computer to approximate the solution for an initial value problem.
“Knowing” that some line works to some ridiculous depth is different than understanding how and why.
And at some level the answer is simply “because every possible refutation fails” and there is no simpler pattern to match against nor intuition to be had. That is the how and why of it.
The scientist can understand “how” the model works, how many layers there are, that each neuron has a weight, that some are connected… Parent comment and yours show that “understanding” is a fuzzy concept.
