Deep Learning's Deep Problem
Posted on Feb. 2, 2018, 9:12 p.m.
In 2010 I started writing a book called Machine Learning in Action, in went to print in May 2012, and has sold tens of thousands of copies in many countries. It was a great experience for me as the author. The book really was based on stuff I had learned as well as what was the state of the art in 2010. One of the pieces of feedback I got was: "maybe you should consider adding deep learning." I ultimately decided to leave the topic out, because there wasn't much code out there at the time, and mostly because I couldn't explain why they worked.
The past summer Ali Rahimi received an award at the NIPS conference and used his time on stage to echo these sentiments. Ali Rahimi's talk at NIPS Ali stresses the point I confronted personally many years ago, most practitioners can't explain why deep learning works.
Over the past few years I have seen numerous attempts to explain why they work, but I feel there is so much hype around deep learning that few stop to ask this question, and it leaves us worse off. So many times when things are going right we stop to ask questions, it's only when things go wrong do most people stop to ask why. This is a common pattern with humans, and one of our biases that prevent us from making good decisions.