> Today we solve the stuff that's possible to solve, the low-hanging fruit of problems. Tomorrow, the next problem we try to tackle is ten times bigger, a hundred times harder, we need machines a thousand times more powerful...
Nah, if you can't handle the huge amount of data, it's possible to just switch to a sparse model or do MC-like sampling. Take AlphaGo as an example for that - a huge state space, yet it was tractable to beat the human expert. That way the network doesn't scale linearly with the size of the domain being learned.
These kinds of solutions don't rely on improving the hardware. What is needed is datasets, competitions and grants to get people to try and improve state-of-the-art results on them. It's been demonstrated that when a good benchmark appears, a lot of papers follow and top results improve massively. Another useful component would be simulators for training agents in RL.
A promising direction is extending neural networks with memory and attention, in order to focus its work more efficiently and to access external knowledge bases. As we improve on these knowledge bases and ontologies, all we have to learn is how to operate on them.
Thus, improvements can come in various ways, by sampling, sparsity, external knowledge bases, better research frameworks, and improving the hardware (such as having a better GPU card or a dedicated device) is just one factor.
>> Nah, if you can't handle the huge amount of data, it's possible to just switch to a sparse model or do MC-like sampling... That way the network doesn't scale linearly with the size of the domain being learned.
That's useful when your domain is finite, like in your example, Go. If you're dealing with a non-finite domain, like language, MC won't save you. When you sample from a huge domain, you eventually get something manageable. When you sample from an infinite domain - you get back an infinite domain.
That's why approximating infinite processes is hard: because you can only approximate infinity with itself. And all the computing power in the world will not save you.
>> It's been demonstrated that when a good benchmark appears, a lot of papers follow and top results improve massively.
Mnyeah, I don't know about that. It's useful to have a motivator but on the other hand the competitions become self-fulfilling prophecies, the datasets come with biases that the real world has no obligation to abide by and the competitors tend to optimise for beating the competition rather than solving the problem per se.
So you read about near-perfect results on a staple dataset, so good that it's meaningless to improve on them - 98.6% or something. Then you wait and wait to see the same results in everyday use, but when the systems are deployed in the real world their performance goes way down, so you have a system that got 99 ish in the staple dataset but 60 ish in production, as many others did before it. What have we gained, in practice? We learned how to beat a competition. That's just a waste of time.
And it's even worse because it distracts everyone, just like you say: the press, researchers, grant money...
Well, OK, I'm not saying the competitions are a waste of time, as such. But overfitting to them is a big problem in practice.
>> A promising direction is extending neural networks with memory and attention
That's what I'm talking about, isn't it? Just raw computing power won't do anything. We need to get smarter. So I'm not disagreeing with you, I'm disagreeing with the tendency to throw a bunch of data at a bunch of GPUs and say we've made progress because the whole thing runs faster. You may run faster on a bike, but you won't outrun a horse.
(Oh dear, now someone's gonna point me to a video of a man on a bike outrunning a horse. Fine, internets. You win).
Nah, if you can't handle the huge amount of data, it's possible to just switch to a sparse model or do MC-like sampling. Take AlphaGo as an example for that - a huge state space, yet it was tractable to beat the human expert. That way the network doesn't scale linearly with the size of the domain being learned.
These kinds of solutions don't rely on improving the hardware. What is needed is datasets, competitions and grants to get people to try and improve state-of-the-art results on them. It's been demonstrated that when a good benchmark appears, a lot of papers follow and top results improve massively. Another useful component would be simulators for training agents in RL.
A promising direction is extending neural networks with memory and attention, in order to focus its work more efficiently and to access external knowledge bases. As we improve on these knowledge bases and ontologies, all we have to learn is how to operate on them.
Thus, improvements can come in various ways, by sampling, sparsity, external knowledge bases, better research frameworks, and improving the hardware (such as having a better GPU card or a dedicated device) is just one factor.