Once you notice it, you can see that this tradeoff is everywhere, and is probably encoded in a mathematical law of nature somewhere *
Robustness vs efficiency. A few examples:
Supply chain. If you store a week's worth of inputs as a buffer to absorb shocks from upstream, but you pay a little bit of extra. A competitor who stores a day will edge you out, at least until there's a supply shock.
Career. If you become a specialist in a narrow niche, you can earn a lot. But if that niche goes away, you have to spend a lot of effort re-training into something general.
Nature. You have plants or animals that are very well adapted to a narrow range of geographic area or weather, they tend to get wiped out when something changes.
Even neural nets themselves. You can overfit and train to the dataset, which lowers your loss and makes you really good at the one task that you're trained at. Or you can miss a few evals on your dataset, but do pretty well at recognizing general stuff that you haven't seen before.
* My personal theory is that in general there is a "price" that you have to pay no matter what. Either you pay it upfront by wasting a bunch of energy in specializing, only to gain efficiency down the road. Or you pay a small hit to efficiency everywhere, but avoid having to waste energy in specialization. It ends up being a wash. I'm sure there is some physical origin to this.
I’ve always thought of it like the law of conservation of energy.
Money is “energy,” your physical and mental health/stamina are, your time is, your knowledge is a store of energy, your connections can be a store of energy, the risk you take can even been seen as an energy in a way.
Then it’s just an optimization problem, there are certainly more and less efficient mixes of these energies, but there is likely a minimum amount of energy you’ll have to expend to reach the goal.
Unlike in engineering, it’s often a lot harder to quantify the values here, so I suspect honing an eye for it is critical to success in life.
Robustness vs efficiency. A few examples:
Supply chain. If you store a week's worth of inputs as a buffer to absorb shocks from upstream, but you pay a little bit of extra. A competitor who stores a day will edge you out, at least until there's a supply shock.
Career. If you become a specialist in a narrow niche, you can earn a lot. But if that niche goes away, you have to spend a lot of effort re-training into something general.
Nature. You have plants or animals that are very well adapted to a narrow range of geographic area or weather, they tend to get wiped out when something changes.
Even neural nets themselves. You can overfit and train to the dataset, which lowers your loss and makes you really good at the one task that you're trained at. Or you can miss a few evals on your dataset, but do pretty well at recognizing general stuff that you haven't seen before.
* My personal theory is that in general there is a "price" that you have to pay no matter what. Either you pay it upfront by wasting a bunch of energy in specializing, only to gain efficiency down the road. Or you pay a small hit to efficiency everywhere, but avoid having to waste energy in specialization. It ends up being a wash. I'm sure there is some physical origin to this.