I logged in to comment on the superb design as well. The design doesn’t only make it look good, every aspect is in full support of the content. And technically it’s executed perfectly as well. Very impressive.
I'm incredibly lucky to be working with a number of people who have an amazing intersection of design skills and scientific knowledge.
Ludwig is fantastic and put an incredible amount of work into polishing this article. And my co-editor Shan (who used to do data vis at the New York Times) seems like he has super powers half the time. We also get lots of outstanding advice from Ian Johnson and Arvind Satyanarayan.
Minor note, you should add the field DOI={10.23915/distill.00007} to the BibTeX citation. This is also missing from Google scholar, and is a particular pat peeve of mine now that DOIs are practically mandatory (copying the helpfully-formatted citation but then having to look around the page to find the DOI).
(In case "I like this metaphor" isn't a comment about how you have liked and continue to like it, it's a reference to an old Buddhist saying. I could have sworn it also appeared on the first page of the Tao Te Ching, but I guess I had just remembered the gist.)
As always, THANK YOU. You and your coauthors' attention to even the smallest details everywhere in this piece are evident. I've added it to my reading list, to read through it and look at the visualizations slowly :-)
That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
> do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)
Thanks. I asked the question in such an open-ended way just to see if you had any crazy ideas. It does sound like a hard problem.
In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?
Many of the visualizations seem interpretable even though they are (essentially, I presume) projections of a very high dimensional space onto 2d position + 3d color space. Is there any research on projecting these visualizations into 6-dimensions as well (3d position + 3d color in VR) or even higher dimensions by substituting other sensory modalities for extra spacial or color dimensions?
I suspect the extra fidelity won't have much payoff for high dimensions considering the large cognitive overhead required for a person to make sense of it. Even a six dimensional approximation of a 1,000 dimensional space is quite crude, and we're pretty confused by spaces larger than three.
Can you elaborate on what you mean by "directions in activation space" ? If I understand it right:
You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct?
> First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.
This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much."
Yep, neural network interpretability research involves quite a bit of staring a pretty pictures. It's a difficult job, but someone has to look at all those images. :P
Great presentation, but I do wish they'd throw in an equation or two. When they talk about the "channel objective", which they describe as "layer_n[:,:,z]", do they mean they are finding parameters that maximize the sum of the activations of RGB values of each channel? I'm not quite sure what the scalar loss function actually is here. I'm assuming some mean. (They discuss a few reduction operators, L_inf, L_2, in the preconditioning part but I don't think it's the same thing?)
The visualizations of image gradients was really fascinating, I never really thought about plotting the gradient of each pixel channel as an image. I take it these gradients are for a particular (and same) random starting value and step size? It's not totally clear.
(I have to say, "second-to-last figure.." again.. cool presentation but being able to say "figure 9" or whatever would be nice. Not everything about traditional publication needs to be thrown out the window.. figure and section numbers are useful for discussion!)
You're right: we are taking the mean of the activations of a given channel `z` over all its `x,y` coordinates. (We could sum, but we use mean so that step sizes are comparable between channel and neuron objectives.)
Thanks for the feedback that this notation is not super clear, we will consider rewriting those expressions.
When we do feature visualization we do start from a random point/noise. For the diagram showing steepest descent directions, however, the gradient is evaluated on an input image from the dataset, shown as the leftmost image.
There's no real step size either as we're showing the direction. You can think of the scale as arbitrary and chosen for appearance.
Ah, thanks for your explanation re the gradient images, I got it, thanks! I think it does say that more or less in the text actually, I was understanding it a bit wrong, my bad. For me this preconditoning part of the article is the hardest to get an intuition for.
Yes that appendix is not to be missed. Also clicking on an image goes deeper into an exhaustive catalog of each layer, which is an amazing and huge resource.
And how can e.g. layer3a be generated from layer conv2d0? By convolving with a linear kernel? Or by the entire Inception Module including the linear and the non-linear operations?
Thank you. Outstanding work breaking it down.
Here's another paper people might enjoy. The author generates an example for "Saxophone," which includes a player... Which is fascinating, bc it implies that our usage of the word in real practice implies a player, even though the Saxophone is an instrument only. This highlights the difference between our denotative language and our experience of language! https://www.auduno.com/2015/07/29/visualizing-googlenet-clas...
Edit: I'll be damned if there isn't something downright Jungian about these prototypes! There are snakes! Man-made objects! Shelter structures! Wheels! Animals! Sexy legs! The connection between snakes and guitar bodies is blowing my mind!
I should have explained how these are made. They train another neural network[1] to produce an image that most maximizes each class. This acts as a prior that the image must have a very simple and regular structure. And so the results seem to be very simple and even abstract, instead of pixel vomit.
[1] Not technically a neural network, but a CPPN. Which is something like a neural network with many different mathematical functions as activation functions. This allows things like a neuron with a sine wave activation that can repeat a pattern across the image.
the school bus one really sticks out to me. it seems all the net cares about is seeing orange juxtaposed with black. No shapes, no vehicle features, just orange and black.
hazarding a guess with no knowledge of the subject, I wonder if that is because no other class in ImageNet can be defined by orange and black. The net simply doesn't need to learn anything more about orange and black, because on 100% of the samples it trained on, orange and black meant "school bus". Every time. So no need to learn any other features -- if you see orange and black, it MUST be a school bus, at least in the context of this data set.
I wonder if we introduced other "orange and black" classes to ImageNet, it would need to learn more features about the school bus in order to identify it.
That's a good observation. But when I see that image I definitely think of a school bus. The color pattern is very distinctive (in the US anyway where all school buses are painted the same color and style.) So I can't say the NN is wrong.
It doesn't necessarily mean that color is the only feature it uses to classify school buses. Just that that feature alone is enough.
would be interesting to try these pictures, the stripes would compete with the other features of the picture, so it might not result in seeing a schoolbus.
Wow. That's incredible how psychedelic these images are. I'd be really curious to learn more about the link between these two seemingly distant subjects.
Neural nets are doing something that the mind does too - it takes visual input, then it "predicts" and "fills in the blanks" (aka hallucinates) so that you "see" what you expect to see. Its why you dont notice your blind spot.
Brain and NNs, at least in the first visual cortex layers and the first perception layers accordingly, are doing the same thing - building optimal sparse coding for the body of imagery they've been exposed too (in general it isn't limited to images, it is just that our knowledge for image processing seems to be a bit more advanced). There are works from 199x proving that Gabor kernels - which form the first layer in the visual cortex and which (or very similar) happen to emerge in the first layers of well trained NNs too - are optimal for a wide class of image inputs (edge density, etc.), basically the class of images naturally surrounding us. That optimality provides for a good explanation of their emergence in NN.
It seems to be a natural speculation that the further layers of visual cortex/brain and NNs are subject to similar optimization, and thus they would have similar kernels. I think psychedelics somehow stimulate these deep layers to massively fire without waiting for the normally required input from the eyes to propagate to those layers to selectively fire neurons in them.
This pictures reminds me about what one's can see under psychedelics. All sensory input basically begins to break down to that kind of patterns, and thus reality dissolves into nothing. This is equally terrifying and liberating depends on look. The terrifying thought is that there's no-one behind this eyes and ears. The liberating thought is that if there's no-one there, then there's no-one to die.
I believe for anybody who has ever tried psychedelics seeing these was a watershed moment. It seems almost impossible that an algorithm could so faithfully reproduce the experience without somehow having recreated some fundamental structure of the human brain. That is compounded by the fact that this wasn't the result of actually trying to do so, but of an open-ended experiment of creating feedback loops in NN layers.
The best explanation I ever heard for psychadelics was that they turn up the gain, the attenuation..on both thoughts and senses. So you get weird stuff which would normally be supressed.
Hi Chris, firstly thanks for all the work you've done publishing brilliant articles on supervised and unsupervised methods and visualisation on your old blog and now in Distill.
This question isn't about feature visualisation, but I though I'd take the chance to ask you, what do you think of Hinton's latest paper and his move away from neural network architectures?
Capsules are cool! I spent several days hanging out with Geoff & co a few months back -- they're exploring really exciting directions. I don't feel like I have anything particularly exciting to add, though.
Interesting that simple optimization ends up with high-frequency noise similar to adversarial attacks on neural nets.
While I agree that the practicality of these visualizations mean that you have to fight against this high-frequency "cheating", I can't help but shake the feeling that what these optimization visualizations are showing us is correct. This is what the neuron responds to, whether you like it or not. Put in another way, the problem doesn't seem to be with the visualization but with the network itself.
Has there been any research in making neural networks that are robust to adversarial examples?
The article mentions pooling layers as one source for the high frequency patterns. Jeffrey Hinton recently introduced capsule networks (https://news.ycombinator.com/item?id=15609402), in part because he wants to get rid of pooling layers. Maybe this approach is effective to counter (at least visually indistinguishable) adversarial examples.
Is there any way to run images from a camera real-time into GoogLeNet?
E.g. like if I want to scan areas around me to see if there are any perspectives in my environment that light up the "snake" neurons or the dog neurons???
Could they train on white noise from a television and see if the CBR shows a structure similar to the structure of the observable universe when examining the feature layers?
Awesome, but to me this stuff is also terrifying, and I can't quite place why.
Something about dissecting intelligence, and the potential that our own minds process things similarly. Creepy how our reality is distilled into these uncanny valley type matrices.
Also, I suspect it says something that these images look like what people report seeing on psychedelic trips...
I remember friends telling me back in 88 [1] about driving round London from one rave to another high on MDMA & LSD. Back then I thought driving on acid was insane. This sounds like driving while the car is on acid! Which is a terrifying thought...
No, the acid simply lets you see "how the sausage is made". You get temporary access to intermediate layers that should probably remain hidden (and for good reasons).
With AI it's trivial to get access to the inner workings of the intermediate layers.
Any insights on how it's generated? Markdown, Rst, Latex -> HTML? I would love to produce my documentation in this way.
Edit: I was too hurried. Everything is explained in https://distill.pub/guide/, the template is at https://github.com/distillpub/template