I think once the underpinnings were created via math it became questions about what network geometries to use.
We don't have much math I'm aware of that can describe the capabilities of different network configurations from first principles. Even though we constructed the network it feels like we're back to the beginnings of science with this one.
The thing is, the underpinnings have been there for decades. The convnet was invented in the '90s. The only thing that's changed is the availability of data and processing power.
ConvNet was indeed invented a long time ago. But a lot of things have changed since then other than data and processing power. There have been new development in optimizers, non linearities, architectures, loss functions and theoretical understanding.
We don't have much math I'm aware of that can describe the capabilities of different network configurations from first principles. Even though we constructed the network it feels like we're back to the beginnings of science with this one.
Let's change this and see what happens.