Once you notice it, you can see that this tradeoff is everywhere, and is probably encoded in a mathematical law of nature somewhere *
Robustness vs efficiency. A few examples:
Supply chain. If you store a week's worth of inputs as a buffer to absorb shocks from upstream, but you pay a little bit of extra. A competitor who stores a day will edge you out, at least until there's a supply shock.
Career. If you become a specialist in a narrow niche, you can earn a lot. But if that niche goes away, you have to spend a lot of effort re-training into something general.
Nature. You have plants or animals that are very well adapted to a narrow range of geographic area or weather, they tend to get wiped out when something changes.
Even neural nets themselves. You can overfit and train to the dataset, which lowers your loss and makes you really good at the one task that you're trained at. Or you can miss a few evals on your dataset, but do pretty well at recognizing general stuff that you haven't seen before.
* My personal theory is that in general there is a "price" that you have to pay no matter what. Either you pay it upfront by wasting a bunch of energy in specializing, only to gain efficiency down the road. Or you pay a small hit to efficiency everywhere, but avoid having to waste energy in specialization. It ends up being a wash. I'm sure there is some physical origin to this.
I’ve always thought of it like the law of conservation of energy.
Money is “energy,” your physical and mental health/stamina are, your time is, your knowledge is a store of energy, your connections can be a store of energy, the risk you take can even been seen as an energy in a way.
Then it’s just an optimization problem, there are certainly more and less efficient mixes of these energies, but there is likely a minimum amount of energy you’ll have to expend to reach the goal.
Unlike in engineering, it’s often a lot harder to quantify the values here, so I suspect honing an eye for it is critical to success in life.
It should go without saying that custom designed chips are going to outperform the more general purpose chips at the task if everything else is equal.
We've seen that with cryptocurrency mining. CPU -> GPU -> ASIC. And many other examples.
Aren't AI GPUs already to a great extent custom designed for AI? Is it even a GPU if it's not used for graphics? What's left on the table in terms of designing an ASIC that specifically targets a single AI task?
Having nicer tools to design and simulate ASICs is cool. I'm guessing that if you work at a higher level you might want to drop in to the lowest levels to extract maximum performance.
If you want to build yourself an order of magnitude of how much performance is left on the table in CPUs, consider the following:
* A single bit Adder with Carry can be made with 9 transistors, and has to wait for 4 stages for the signal to propagate into. Then you have to stack 32 of them in series for a 32 bit carry. And wait for the signal to propagate.
* Then you first have to route the adding instruction, which is code treated as data. To do a single addition, you have a whole meta pipeline to run. With a lot of side-support: prefetching, cache coherency, loading, writing, instruction pointer, stack pointer, etc.
* Then you have to have the data available, or else you'll be waiting for RAM. That can be as long as 200 cycles, all the while you could be doing several additions in parallel; wasting 650 picoJoules, when the addition in itself is just a few picoJoules.
As silicon scaling laws hit their respective walls (latency, clock limits have been hit; bandwidth, memory sizes remain growing) the only way we'll be able to eek out more performance is to go ASIC on a lot of stable code.
Robustness vs efficiency. A few examples:
Supply chain. If you store a week's worth of inputs as a buffer to absorb shocks from upstream, but you pay a little bit of extra. A competitor who stores a day will edge you out, at least until there's a supply shock.
Career. If you become a specialist in a narrow niche, you can earn a lot. But if that niche goes away, you have to spend a lot of effort re-training into something general.
Nature. You have plants or animals that are very well adapted to a narrow range of geographic area or weather, they tend to get wiped out when something changes.
Even neural nets themselves. You can overfit and train to the dataset, which lowers your loss and makes you really good at the one task that you're trained at. Or you can miss a few evals on your dataset, but do pretty well at recognizing general stuff that you haven't seen before.
* My personal theory is that in general there is a "price" that you have to pay no matter what. Either you pay it upfront by wasting a bunch of energy in specializing, only to gain efficiency down the road. Or you pay a small hit to efficiency everywhere, but avoid having to waste energy in specialization. It ends up being a wash. I'm sure there is some physical origin to this.