As I mention in my comment [1], there are already a lot of designs that do work on very real world problems. Also, we've already started mapping CNNs onto the silicon for greater power efficiency. But you're right to say there's still a lot of work to do.

[1] https://news.ycombinator.com/item?id=14084846