I spent a few weeks closely reading this book and I have to disagree with the majority here. I didn't like the book at all. And I am an advanced math geek.
My main issue is that the book tells you all about the different parameter tweaks, but passes little concrete wisdom to the reader. It doesn't distinguish between modeling assumptions, and it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of.
I think it boils down to something that I have been feeling and hearing a lot in the past few years: the statistical jargon is so overwhelming that the authors can't explain things clearly. I can point to many examples in this book that I feel are unnecessary stumbling blocks, but the fact is that I'll spend an hour or two discussing parts of this book with a room full of smart machine learning researchers, and at the end we'll all agree we don't understand the material better than we did at the start.
On the other hand, I'll read research papers that don't force the statistical perspective down the reader's throat (e.g. http://arxiv.org/abs/1602.04485v1) and find them very easy to understand by comparison.
It might be a cultural difference, but I've heard this complaint enough from experts who straddle both sides of the computational/statistical machine learning divide that I don't think it's just me.
>> it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of
It is good to see critical views but it will be even better if you could give concrete examples for statements like the above. Also, what other books do you recommend?
This has been my frustration. I've been wanting to grok deep learning, but haven't found any source that can explain it in a way that doesn't overcomplicate simple things (which I can only tell it's doing when I already understand the topic). I also don't have the time or incentive to really dig in and do the math myself from scratch, since so much of it is wide open research directions.
I've also had this experience with much much simpler areas of statistics and statistical ML (cf. Markov chain monte carlo), so this is a sort of recurring theme for me. Considering how simple MCMC is now that I do understand it, it's difficult to dispel the nagging feeling that all the statistical ML literature is (likely unintentionally) obfuscated.
I could give concrete examples of excerpts and entire sections of the book that don't make sense to me, but I don't think it's all that productive because a lot of it boils down to organizational disagreements, cultural assumptions behind the math, and differences in priorities. Individually it seems like nitpicking, but they add up quickly to general muddled confusion. This is especially true when, for example, almost always the right answer to the question "Why does this particular technique work?" is, "We have no clue, but here is some anecdotal evidence and half-substantiated oversimplified theories." Instead these answers are passed as well-known fact.
Have you read the acclaimed "Elements of Statistical Learning"[1] or its lighter weight counterpart, "Intro to Statistical Learning"[2]?
I think both explain statistical ML clearly at just the right level of detail. But I'm not an advanced math person, so it's certainly possible that I've failed to notice any flaws in their approach.
I haven't looked too deeply at this particular book yet, but your overall sentiment is sounds very familiar.
Too many people are willing to accept (and defend!) sub-par explanations and overcome them through titanic mental grind. The scary part of it is that often this leaves you with a broken mental model that continues to require tons of effort in application. Meanwhile, much better explanations exist. They make learning easy and fun, while also giving you intuition that is easy to apply and even expand to other areas.
I am pretty good at spotting bad mental models, but it's really hard to prove that they are bad, until you find one clearly superior. Recently I stumbled upon MIT's linear algebra class at Open Courseware and was blown away by how much easier it was to follow than the stuff I had in college, while covering the same material in much greater detail.
The new O'Reilly book "Fundamentals of Deep Learning" by Nikhil Buduma (available on Safari for a while now) is good at the fundamentals- very clearly explained, nice diagrams. It it relatively close to the path of my Neural Networks classes (although those were 20 years ago).
It necessarily shorter on detail in terms of the tricks of implementation that have radically improved performance of these techniques over the past 5 years, and might serve as a good read before diving into this via Goodfellow, Bengio, and Courville.
It is valuable to cite examples because it will help others determine if they have the same organizational disagreements and cultural assumptions that would make the book burdensome to read.
On the other hand, if we find the cited examples clear due to e.g. different backgrounds, expectations, perspectives, etc, then that might be a good signal that we should look into the book.
You know, if you wanted to write a book, you could set up a Patreon with a per-chapter benefit. Perhaps write the introduction for free, setting out your goals for the book. I'd chip in.
Even if the material is already 'covered' elsewhere, simply re-explaining the same concepts would do a world of good for those who think in the same way as you.
If anything, to me a lot of deep learning literature seems to lack the statistical insight and theory that's available to other subfields in machine learning (whether the Bayesian/graphical models camp, the statistical learning theory camp...)
If this book is trying to do more to bring statistical or probabilistic insights to bear on deep learning than I think that's a very good thing. It might make it less accessible to those coming from a pure computer science background, but potentially more so to those who like to think about machine learning from a probabilistic modelling perspective.
If they're using stats jargon in a gratuitous way that doesn't actually cast any light on the material then that's another thing, but from a quick skim I didn't see anything particularly bad on this front. Do you have any examples of the kind of jargon you're talking about?
To others reading, I just wanted to emphasise that statistics is really important in machine learning! Deep learning lets you get away with less of it than you might need elsewhere, but that doesn't mean one can treat it as an unnecessary inconvenience. It's a language you need to learn, especially if you want to try and get to the bottom of how and why aspects of deep learning work the way they do. As opposed to just an empirical "using GPU clusters to throw lots of clever shit at the wall and see what sticks" engineering field. Bengio seems very interested in these kinds of questions and I'm glad he's leading research in that direction, even if clear answers and intuition aren't always easy to come by at this point.
I, too, spent some time reading this book, and arrived at the conclusion that it's terribly written.
From a mathematical perspective, the book does not define hardly anything. Take, for example, the important chapter on Deep Feedforward Networks. They hem and haw saying things like "quintessential deep learning models" but never actually explain _what a deep feedforward network is_. There are no definitions: they give what amounts to preamble of a definition, and then move right on to using the notion, but don't actually give the definition itself. In the deep feedforward network example, the closest thing to a definition is, "A feedforward network defines a mapping y=f(x;theta)". That doesn't really say anything though. If you're asked to define "dog", you can say, "A dog has four legs", but how does the reader know that a cat isn't a dog?
I find that for basic machine learning it's much better to just go through machine learning framework tutorials. For advanced learning, I'm not there yet so I can't speak from experience but I'll be shocked if this book fits well in any role except maybe last-resort reference material. If that.
Part of the problem in writing a deep learning book, is that very little that warrants being in a book, is actually known. The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models. Consequently, that line of research is also under-cited and under-developed. In some ways, 'deep learning' is in a different Kuhnian paradigm altogether. So, people who are used to learning about all the intricacies of classical ML models fail to appreciate deep learning, because the metrics with which you would judge a deep learning model are different from the ones you would use to judge a classical ML model. The same applies for the books in different Kuhnian paradigm.
>The mainstream deep learning academic community welcomes theoretical work but articles on new techniques which beat SOTA are given much more attention than articles on setting up a theoretical structure of the models.
The fact that beating an accuracy test on some arbitrary data set is seen as more important than understanding why and when your methods works (or do not work) is deeply disturbing.
It's not an arbitrary dataset. To be taken seriously you need to beat SOTA on several standard benchmark datasets where only reaching SOTA on one is already hard.
Or alternatively, reach comparable level of validation accuracy with significantly less compute or memory usage (computational complexity).
Or alternatively, reach comparable level of validation accuracy with only a subset of labeled samples + unlabeled samples (sample complexity).
...which is very valuable. After all, we don't know when, if ever, a satisfactory explanation (by what criteria?) will derive. To defer all progress in the field until it should meet the standards of some other field would be a great loss, and might preclude any number of future further discoveries.
As far as I understand the concept of "science", an empirical observation (including "it works from an engineering perspective, kinda") is only ever an observation. Scientific progress, by definition, involves advancing our understanding of hows and whys.
Except for conference papers, which are even worse. There's so much "secret sauce", only few are independently reproducible. Sadly, in some research areas they are the dominant way of publishing research output.
You still should, I think. I mean, at least give it a try. Different people can have completely different experiences when reading the same material. Perhaps you will love it, perhaps you will agree with the OP. In any event, it's worth getting it straight from the horse's mouth.
I agree. Everyone should give it a try if they're genuinely interested in learning it. But if you find yourself struggling with it, know you're not the only one :)
Statistics [edit: and especially ML] is different from math. It is important to let go of one's preconceived notions and to try to absorb the new field as is. Its jargon exists for a reason --- the normal math jargon doesn't quite have the right concepts --- so it is worthwhile spending the effort to understand it.
Yeah, but I did my PhD in theoretical computer science and have published research papers in ML. So I shouldn't have such a hard time. (It could just be me, but maybe it's a weak signal)
1. It also covers "classical" artificial neural networks, i.e., things like backprop from before Hinton and others made breakthroughs for deep learning. This means you can start with this book even if you are new to ANNs. The later sections cover "real deep learning".
2. The language is great for beginners and users. You don't have to be an advanced math geek to follow everything. They seem to cover a fair amount of ground too, so its not dumbed down either.
3. I guess it covers most of the underlying theory and practical technicques but is implementation neutral. You should probably pick up a tutorial for your favorite implementation like Theano, TensorFlow, etc.
The old backprop from the 80's / 90's is still in use and the primary way to train deep nets. We tend to call it SGD (on a composition of differentiable functions) nowadays but it's the same algorithm.
I'm working my way through that at the moment and so far it's pitched rather well (at least for my prior experience), describing things simply and concisely but still with enough background that it's easy enough to see where everything fits.
Unfortunately, that's just the ToC and the bibliography. It looks like their "contract with MIT Press forbids distribution of too easily copied electronic formats of the book."
Does that mean there's a fifteen-step asciidoc/gitbook/LaTeX build that is freely distributed? Because I'm willing to spend a few minutes building a pdf for the sake of getting good content.
In case anyone else has trouble with the "Save as PDF" part, I did it successfully using Firefox on OSX: from the print menu, choose "show details" and change all headers/footers to "--blank--" so there's no ugly URLs/dates in the corners, then press PDF -> "Save as PDF". (In Chrome, on the other hand, saving/printing as PDF chopped each page into quarters for me...) At least in the one chapter I've saved so far, all the math notation renders just like in the HTML version.
That would give you a PDF version of the web page, right? Wouldn't you still have problems with "some things like subscript expressions do not render correctly"?
Doubt it. We talked to MIT Press about publishing our book and they were extremely hostile to having any way for our readers to produce an un-DRM'd PDF, which is how our book is currently distributed.
Actually, they were mostly just very hostile and unpleasant. This is among the many reasons why we're probably staying independent and printing the book ourselves - so that the digital version stays unencumbered among other things.
I don't claim to have a solution, but these models of book monetisation really seem doomed. What are the chances that I will buy this book just because they made it artificially harder for me to download it? Probably a net negative.
Just saying, but if you want to hop onto the ML bandwagon (for instance), then don't bother going over linear algebra or probabilities first, and instead just learn what you need as you go. For example, the first sections of this book are already devoted to getting you on the right track, and it's somewhat standard to do so. And besides, there's no need in learning what are rotation matrices if you won't use them.
As a counterpoint, if parent is interested in taking ML further, a solid foundation in linear algebra will be huge when more advanced signal processing applications come up.
There isn't much hard probability/statistics in deep learning since most of the stuff is empirical(IE: these neural network architectures work b/c we tried it and it works!). I'd say that deep learning is very accessible to the novice practictioner if you really want to dive into it; it doesn't require the same sort of mathematical background that something like signal processing might.
For linear algebra I heard Paul Dawkins' document is great, it's not on his site anymore but you can find it online. I've read calculus 1 and 1/3 of calculus 2 and it's good material.
This looks great, any other recommendations for enjoyable reads on ml/stat learning?
ESL
ISLR
doing Bayesian data analysis w/ jags/Stan
bda3 - gelman
prob graphical models
Convex analysis - Boyd
adv data analysis from elem pov - shalizi
Trying to build out my library. I have a background in prob/stats/analysis and measure theory/linear algebra and also knowledge of algorithms and data structures at the advanced undergrad level, So I'm not too concerned about technical depth just want to enjoy a good technical expository and gain intuition.
> Printing seems to work best printing directly from the browser, using Chrome. Other browsers do not work as well. In particular, the Edge browser displays the "does not equal" sign as the "equals" sign in some cases.
Of all the printing bugs for a maths/logic heavy text! Can just visualise the head banging against the desk upon discovering this one, having struggled with understanding something for hours - well ok, either the universe is broken or... oohhhhhhh :-D
Which browser are you using? It is perfectly readable in Safari and Firefox. Printing to PDF from Firefox on OSX yields excellent results, while printing to PDF from Safari didn't work well for me.
Can any practitioners / experts out there comment on the range of topics? For example, I understand the book to be introductory, and so the scope is likely somewhat limited. But how close does it get you to the ANNs currently in use, at least conceptually if not in complete detail? Thanks!
Non-expert DL practitioner here. The coverage is very good - if I were running some kind of "Deep Learning onboarding class" for a university or tech company, this is what I would present.
My favorite aspect of this book is that it provides a graphical models interpretation of DL methods, which is the most powerful perspective we have right now to reason about model design (instead of some large black box function that we train end-to-end without knowing what's in between).
It also explains some fairly recent models and techniques well (VAE, DCGAN, regularization) that form the basis of more complex architectures. If you understand the models here, you should be able to understand the design choices made in more complex architectures.
Thanks to Goodfellow, Bengio, and Courville for this excellent work.
"This format is a sort of weak DRM required by our contract
with MIT Press. It's intended to discourage unauthorized
copying/editing of the book. Unfortunately, the conversion
from PDF to HTML is not perfect, and some things like
subscript expressions do not render correctly. If you have a
suggestion for a better way of making the book available to
a wide audience while preventing unauthorized copies, please
let us know."
(Obligatory: http://www.xkcd.com/974/ -- but Knuth solved the arbitrary condiments problem, and others besides.)
The Concrete Mathematics book that resulted was a masterpiece in my opinion. I learned so much about how to computationally do discrete math from that book. And it's a very elegant package.
It's amazing to me that he designed a program (METAFONT) to design fonts for him, and that the various outputs of this program actually are pretty good fonts.
spent lot of time to convert each pages using this website and merged the pdfs into single pdf and then found that Athena has not done the conversion correctly. The diagrams not converted correctly.
Don't quite get the complaints about it not being available in PDF. "We'll publish your book, and you can give it away for free as long as you make people click through to each chapter" is a much, much better deal than I would expect from a big publisher.
Pretty much ... the only edge case I feel for is having an offline copy in places with low or expensive internet access. Like parts of the developing world. Clicking through isn't such a big deal.
In the old days IE had a feature for saving pages for offline reading that would ask you how deep to recursively traverse. It was considered hilarious to set it for 99 on the school dialup connection.
edit: changed ".pdf" from a slightly longer approach to ".html" which actually exists in this workflow. Thanks @TheCabin!
edit(2): ... and gave a valid path to this listdir. Check before I post... check before I post...
edit(3): ... and removed the os.listdir line no longer needed in this approach. Gosh. Just ignore what I was saying and build your own approach. That'll probably be faster at this rate.
Avoids the awkward file system structure by using pdfkit.from_url, but creates one .pdf for each chapter. I tried using a list of urls, but pdfkit failed because my version of wkhtmltopdf did not accept multiple input files.
Edit: pdfkit.from_file also fails on my system when passing multiple files. If that works for you, multiple urls are probably fine, too.
Absolutely right. That's erroneously copied from converting each HTML file separately with the intention to merge afterwards. Thank you for pointing that out!
For what it's worth I got pretty decent conversion results of one file at a time with the Ubuntu repository version (0.9.9)
But that version doesn't work with collecting it into a single output file, no.
I think it was a mistake to try shaving off a couple of lines and a step at the price of a more convoluted install and a brittle process.
Most people would probably be best off just converting one HTML file at a time to pdf (e.g. pdfkit.from_file(filename_in_html,filename_in_html[:-4]+".pdf") in some sort of iteration over the file names) and then concatenating the resulting PDF:s, for instance from command line by
Look, I'm not planning on printing the whole thing, binding it and sticking it on my shelf. I want the pdf because then I can use it when I'm offline, across devices and search it easily.
If I want this book in hard copy, then I will purchase it - I've done this regularly with free digital books - but when it is offered free digitally then in my opinion prohibiting to only certain file formats is futile (as evidenced here), and such constraints are ineffective attempts to encourage people to buy the hard copy through inconvenience.
And I must add that this is no slight to the authors, whom have my greatest appreciation for compiling their vast knowledge into a book and offering it for free. These guys are legends.
I guess this could be automated. For instance, you could download all html files using a plugin like "Download Them All" with a renaming mask like "inum-nameinum.ext" and then try:
There are also tools to convert the resulting files to a single pdf. The only problem I got is, that the woff fonts are not rendered by "wkhtmltopdf" :-/ Ideas?
If only we could convince publishers to publish GFDL or Creative Commons licenses the way we've convinced some software companies to commercialize open source software...
My main issue is that the book tells you all about the different parameter tweaks, but passes little concrete wisdom to the reader. It doesn't distinguish between modeling assumptions, and it replaces very simple explanations of concepts with complicated paragraphs that I can't make sense of.
I think it boils down to something that I have been feeling and hearing a lot in the past few years: the statistical jargon is so overwhelming that the authors can't explain things clearly. I can point to many examples in this book that I feel are unnecessary stumbling blocks, but the fact is that I'll spend an hour or two discussing parts of this book with a room full of smart machine learning researchers, and at the end we'll all agree we don't understand the material better than we did at the start.
On the other hand, I'll read research papers that don't force the statistical perspective down the reader's throat (e.g. http://arxiv.org/abs/1602.04485v1) and find them very easy to understand by comparison.
It might be a cultural difference, but I've heard this complaint enough from experts who straddle both sides of the computational/statistical machine learning divide that I don't think it's just me.