I agree it would be good to improve at weighing trade-offs. This is the standard method of progress in many fields, systemization of data, the finding of rules, the development of models - theories, in short. A nice practical example is Deming's quality management. They used to think you could improve quality up to a certain point, but beyond that you just have to put up with some random amount of defects. He showed that with measurement and accurate models, you can increase quality as much as you like. Is that more or less your thinking?

But I think there's a fundamental barrier for software, because programs can differ from one another arbitrarily. This makes it hard to generalize - and generalization is the essence of theory, rules, systemization, modeling etc. I assume you'll accept that generalization is crucial, but you'll disagree that software can vary arbitrarily.

The reason I say this is because if you find regularity in software... you can automate it. Factor out a common method. Extract a "pattern" into a framework (or better, in functional programming). Derive one thing from another. It's like compression: as soon as you notice something that you can generalize, and construct a theory to model it... you can use that theory to eliminate the redundancy.

In a deeper sense, programming itself represents theories. And theories (e.g. of physics) are not generalizable: we don't have a theory of theories, that enables us to automatically generate the next correct theory. Oh, we can automatically generate theories, just not correct ones. For that, we need data to test them, to find out which one is correct. And so the unknown is the ultimate cause of the problem: we can't generalize what we don't know.

The best you can do is come up with rules of thumb, that help in specific domains - but they only work there. And if they are really accurate, someone will encapsulate them in a method, a framework, a library, a language or operating system, so that programmers don't need to deal with them, and that pattern disappears from their programs.

And to finish off with one of my favourite quotes, by Alan Turing: There need be no real danger of [programming] ever becoming a drudge, for any processes that are quite mechanical may be turned over to the machine itself.

NB: in practice, there is an enormous amount of repetition in programs (and parts that are common between different programs). This amounts to ongoing demand for libraries, frameworks, languages, operating systems, and extra features for them.

So, it's hard to obtain a precise weighing of the tradeoffs for clarify, efficiency, expediency when the playing field keeps changing. We can get better, but not much. It's different in fields like civil engineering, where materials and techniques don't change often. BTW Another fuel for change in software is Moore's Law - which enables different techniques to become favoured that weren't plausible before (like dynamic languages, VMs, etc).

sorry, such a long and self-indulgent response. Maybe it doesn't apply so strongly to trade-offs (though I think clarity is closely related to theory). At any rate, it doesn't stop us improving, which is what you suggest.