I think the author is overstating the importance of being able to explain in human terms decisions made by a neural network. For instance , there is no one reason that I am able to recognize a dog as such. Any feature or combination of features I can think of can be had in another animal. Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me.
The examples normally given for wildly inaccurate predictions were concocted by training a separate neural network to trick the original neural network which seems be just showcasing the effectiveness of neural networks rather than highlighting a weakness.
Also, I would note that human intuition is not immune to tricks. For instance optical illusions regularly trick our perception.
No, the importance is massive, and you understate it. While we may all want to just believe, the regulatory ecosystem worldwide (where it applies, esp. in financial realm) demands that we provide explanations of why models made certain decisions on certain data. Without this ability, models will not be allowed to drive innovation or decisions in many areas of life, from financial (credit and risk) to medical (recommendations for treatment) to legal (best contract approach or best defense approach for a lawsuit).
Saying that humans make mistakes and cannot explain their decisions is, in fact, one of the very reasons we want to have better models. We hope they will do better than most people to create a better world. And their explanations will hopefully provide insight into how we as people make good (and bad) decisions.
And sure, we can change the laws over time, but having advanced models which can allow humans to understand decisions and even provide diagnostics to improve the models will be transformative. Until then, we will see massive impact in some areas of our lives, and frustrating holdbacks in others, driven either by the distraction of building for regulatory constraints or by choosing not to build in regulated areas at all.
We know why these models make decisions on data. They're optimizing for lower error rates.
The black box unveiled for a convolutional neural network is this: over the course of several thousand rounds of performing the dot product of the RGB values of a given training datum against a weight vector, this network has determined that a weight vector containing these values optimally reduce the error produced by "squashing" the output matrices of said dot products in a softmax function when the "squashed" value is compared against the pre-determined true value.
If you would also like to be able to correctly predict whether a given input is part of the class for which this model was optimized for, we suggest your weight vectors also contain these values as this will reduce the number of false positives and false negatives you will produce from your prediction.
No. If the machine is telling me what medicine to administer to the patient, then I want to know exactly what data points the machine thought were relevant (i.e., the relevant symptoms). Furthermore, I want to know what about those symptoms indicated to the machine a particular diagnosis.
The criticism of artificial intelligence in making decisions in lieu of human judgement often pits the process against an impossible ideal that doesn't exist and likely could not exist.
For instance, some research suggests that judges are a lot less likely to grant parole before lunch. The theory is that no parole would be the default no-brainer decision a judge can make, which makes sense. [0]
The fact is that many decisions are made by humans with, at best slight personal bias and at worst malice.
With a systematic AI approach you can be somewhat confident that the decisions made are based on a wide range of experience, optimizing for an agreed upon objective (e.g. likelihood of reoffense). It's not perfect but it's better and can improve a lot easier without the need to villainize individuals
>Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me.
There is an interesting experiment you can do to directly demonstrate this to yourself
From Julian Jaynes' The Origin of Consciousness in the Breakdown of the Bicameral Mind
> Take any two unequal objects, such as a pen and pencil or two unequally filled glasses of water, and place them on the desk in front of you. Then, partly closing your eyes to increase your attention to the task, pick up each one with the thumb and forefinger and judge which is heavier. Now introspect on everything you are doing. You will find yourself conscious of the feel of the objects against the skin of your fingers, conscious of the slight downward pressure as you feel the weight of each, conscious of any protuberances on the sides of the objects, and so forth. And now the actual judging of which is heavier. Where is that? Lo! the very act of judgment that one object is heavier than the other is not conscious. It is somehow just given to you by your nervous system. If we call that process of judgment thinking, we are finding that such thinking is not conscious at all. A simple experiment, yes, but extremely important. It demolishes at once the entire tradition that such thought processes are the structure of the conscious mind.
The DARPA proposal the authors are working on is driven by the military. Consider the applications of an AI whose job it is to find hidden weapons caches from aerial photography of a city, or likely location for IEDs along a road.
You could train a NN to do either, given lots of training data, but before risking the lives of special forces, and potential for collateral damage, you might want to question the AI as to why they think what they found was a hidden cache of weapons or roadside IED.
I agree the statement "humans are surprisingly good at explaining their decisions" is dubious at best. Most explanations are a matter of what's called reconstructive cognition, which is far from accurate in explaining the reasons for an action.
>For instance , there is no one reason that I am able to recognize a dog as such. Any feature or combination of features I can think of can be had in another animal. Something deeper is happening when I am able to correctly identify dogs that is unexplainable, at least by me.
Well, your categorization judgement is probabilistic. Some features make "dogness" more likely, some less likely. However, there is in fact a specific causal structure which constitutes dogness (ie: organism grown via natural reproduction along the canid species-history, with the accompanying genotype and within normal phenotypical variation). Your brain tries to learn a causal model which approximates the correct causal structure, given the evidence available to your senses.
Better: the human brain also seems to perform second-order inference, reasoning about which models predict most precisely, what's relevant to what, and which latent parameters in the learned causal model are most relevant.
So when you see a dog-like tail versus a cat-like tail, you can identify which apparent features are making you say "dog" or "cat" most strongly, which ones are the strongest evidence.
Explanation is hugely important, and the lack of it was one of the factors that killed NNs in the 90s. The ability to explain was also one of the reasons rule-based systems succeeded as well as they did, despite having huge disadvantages vs NNs for many problem domains.
AI for high-consequence decision-making is going to need the automatic feature extraction and robustness of NNs coupled with the explanatory transparency of expert systems. That's been the holy grail for almost 40 years.
Part of it is they want to know that the AI isn't discriminating in illegal ways (eg using race as a feature).
I also worked with the data science guys at a telecom in a past startup and they were so used to using white-box models that the thought of not having a decision tree or other means of explaining how the results came about scared them, just because they weren't used to it and they could check it for correctness etc. Not having that was scary to them.
Another way to think of it is, science vs. magic/customs. Indeed empirical knowledge can provide reasonable answers (e.g. when rubbing a particular mold over your injury it will heal quicker). On the other hand, understanding what an antibiotic is, how to properly dose it and potentially synthesize it from other sources, etc. is a far better tool for the job.
I suspect reality is a messy mix of science and magic/customs, and perhaps one way in which AI can help us is to figure out - within limits -
where and when to take the shortcuts (magic/customs) and when/where not to.
But of course, there are bound to be all sorts of weird unintended consequences when we let what is essentially an alien life-form come up with these things.
I think the gorilla/black people issue is a perfect example of this. An 'AI' might group gorillas and black people together because its 'magic' made it do so. On the one hand it might seem like a rather unfortunate 'mistake', but it's also not difficult to understand why this would happen. I've had more than one experience with white children causing embarrassment by comparing black people to gorillas/monkeys.
I'm not going anywhere in particular with this other than that 1) I don't think science and magic/customs are quite as easily disentangled in daily life, and 2) I have high hopes for the role of AI in all aspects of human development, but I expect it to cause all kind of issues that, in hindsight, we might write about in the same way we write about culture clashes or hypothetical alien visitations. Until we create AI that simulates or experiences actual human lifespans, they'll be basically aliens to us in all kinds of ways, small and big.
The examples normally given for wildly inaccurate predictions were concocted by training a separate neural network to trick the original neural network which seems be just showcasing the effectiveness of neural networks rather than highlighting a weakness.
Also, I would note that human intuition is not immune to tricks. For instance optical illusions regularly trick our perception.