I don't think we need to protect people from learning internals, they do that just fine on their own. I know many situations where we failed to understand internals even in the presence of compelling reason.

With apologies for continuing to quote myself:

"I want to carve steps into the wall of the learning process. Most programs today yield insight only after days or weeks of unrewarded effort. I want an hour of reward for an hour (or three) of effort." -- http://akkartik.name/about

I talk occasionally about viewing your code from the debugger but that is kind of hand wavy. I wonder if there’s a ‘linter’ one could write that looks at coverage or trace reports and complains about bad patterns.

The problem is, we have been adding layers of indirection for 80 years, and each is leaky. So now it's very difficult to do basic stuff, and people are ok with software thick with abstractions.

The next stage should be removing unnecessary layers of indirection, as, like you said, things are much easier to understand and maintain that way.

"All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection." -- David Wheeler

But we don't have 900 layers or anything. Performance is... usually pretty darn good, except for uneccessary disk writes.

It's trivial to do basic stuff because of those layers. What's hard is doing low level things from scratch in a way that doesn't conflict. But at the same time, there's less and less need for that.

I think things are way easier to maintain with a bunch of layers than the old school ravioli code. Most layers are specifically meant to make it easier to maintain, or they just formalize a layer that was already effectively there, but was built into another layer in an ad hoc manner.

Pasta-based metaphors are the best.

However there are all sorts of little subtle costs in your system that aren't captured by most of these tools due to lack of resolution (I haven't spent a lot of time with Intel's hardware solution) and asynchronous costs like memory defragmentation. Depth and frequency of calls are a useful proxy for figuring out what rocks to look under next after you've exhausted the first dozen. For this reason the flame graph is a useful fiction so I don't poo-poo them where anyone can hear. I can barely get people to look at perf data as it is.

But then I think how I'm turned off by some fictions and avoid certain fields, like parser writing, and wonder if a more accurate model would get more people to engage.

But those two things are never going to be perfectly aligned. Enjoyment of the process does not guarantee working results that people want to pay for, and the thing that people want to pay for may be deeply unenjoyable. You can choose to tackle a subset of the problem space that happens to be both enjoyable and worthwhile, but you ought to admit that you are looking at a subset of the problem space.

I do a lot of other things with my time that are also near the confluence of enjoyable and marketable. I sing, for instance, and choir rehearsal and individual vocal practice are both a lot of fun. But any professional musician will tell you that if you want to be world-class at it, you occasionally have to wring the fun out of it. I've chosen the other option: I'm a volunteer singer with my church choir. I sing because it is fun, and only insofar as it is fun. We do new and not-very-difficult music each week, and an hour of effort is inherently an hour of reward. If my choir director said we're going to spend six months drilling some difficult measures and perfecting our German pronunciation, we'd all leave.

(In that sense, a volunteer church choir is rather like hobbyist open-source development. If it produces an enjoyable result for the public, great, and we do find that rewarding. But there is an upper bound on just how hard we're going to try and on the complexity of what we commit to doing.)

If you want an hour of reward after an hour of programming effort, that's fine! But the employment contract I've signed promises a year of payment after a year of effort. Occasionally, I want to work on things that are immediately rewarding because that helps with motivation. And it's important that such work is available. But I have a job that ultimately demands specific products, some of which just aren't going to be immediately rewarding, some of which do actually take a lengthy learning process, and - most importantly - many of which require building on top of other people's work that I could understand if I wanted but there is no business value in doing so up front.

(Incidentally, in the other direction, there are cases where I want to tackle problems that promise years of reward only after multiple years of effort, and figuring out how to get time to work on those problems is actually hard, too.)

We share almost all our code internally and leverage lots of open-source code so that we all have the option of understanding the code if we need it - but we rely on division of labor so that we have the option of building something on top of the existing base today, if we need that, which is more often what we need.

If you want to work on a hobbyist UNIX-like kernel for enjoyment's sake, great, I'm genuinely happy for you. But my servers at work are going to keep running a kernel with millions of lines of code I've never read, because I need those servers to work now.

"Most programs today yield insight only after days or weeks of unrewarded effort. I want an hour of reward for an hour (or three) of effort."

So they want to be rewarded with at least an hour of saved time, or more productive time, for one (or few) hours they put in, instead of spending weeks to reap rewards.

This is more the gist of the article than "lets not have libraries or abstractions at all", its pointing more towards "lets defer those questions until the right time rather than from the jump".

I think Enterprise Java and the insane complexity of modern web dev are indicative of a consequence.

>Because... whenever someone tries an alternative approach, there usually seems to be the hidden unspoken assumptions that they actually don't want software to be as big as it is.

I really don't think that OP is making this assumption

Anyone who has ever done repair work can tell you how often people wait until it’s far too late to do basic maintenance. Software is similarly bimodal. People either run it till it falls apart, or they baby it like a collector’s item.

We have not collectively learned the trick of stopping at the first signs of pain or “strange noises” and figuring out what’s going on. But mostly we have not learned the trick of insisting to the business that we are doing our job when we do this, instead of plowing on blindly for six to eighteen months until there’s a break in the schedule. By which time everything’s a mess and two of the people you hoped would help you clean it up have already quit.

I save myself a lot of frustrating work by stopping for a cup of coffee or while brushing my teeth or washing my hair. “If I change Z I don’t need to touch X or Y”

It's because time pressures are designed in a way that incentivizes you not to do maintenance when you see a mess. It's also not sexy on a resume that you altered something from a mess to not a mess vs implementing some new feature.

It takes highly detail oriented and creative people to develop good software and those traits tend to drive one crazy to the point of fixing something when you see a mess. Given no constraints I bet most developers would clean up their code to the best of their ability and fix things as they come across issues they identify. I've been in these no constraint environments, usually on stuff I write myself for myself, and I don't mind going back and doing a significant refactor when it's clearly needed. Once I'm done, I feel genuinely satisfied that I've done something useful and productive because I only need to convince myself and because no real external financial pressures exist in this context.

Slow is smooth, smooth is fast. We spend a lot of energy running as fast as we can in the wrong direction.

In fact, if put in plainer terms, every engineer would nod in agreement with Hyrum's Law. Everybody has a story of a design that worked because of an undocumented property of a material, that stopped working when Purchasing changed to a new supplier of that material.

Some bright-eyed purchasing agent substituted bark-pellet cat litter, because it was ecological or something -- anyway, not cheaper -- resulting in need for a cleanup so expensive that how much it cost is classified.

A civil engineer is going to become an expert material scientist, "on need"? I doubt it.

It's even widely accepted to be an important part of the education.

Just as Karnaugh maps and parser theory is part of computer science engineering curriculums. It's not something that's expected to be used a daily basis but some general knowledge of how a processor works is absolutely necessary to be able to at least talk about cache line misses, performance counters, mode swaps etc.

As a result, the industry settles on an "efficient" way of managing issues that require depth or breadth, which has to have a few people around who handle those things when needed. That becomes a form of division-of-labor.

I was merely reacting to the statement that most construction engineeers wouldn't know "how concrete is made". Most of them could tell you a thing or two about it, it's even in the curriculum. They are even expected to know about different preparations.

The idea that specialization doesn't exist is a bit of a straw man argument and not something anyone seems to argue.

Or you're solving your current problems in the most efficient means possible, e.g. engineering. Minimizing one challenge frees mental processing power to worry about bigger issues. I couldn't care less how the memory paging in my OS works. I care about building features that users find valuable.

Division of labor ensures that changes can happen reliably and quickly. I’m ramping on an internal library right now and it’s taking a while to understand it’s complexities and history. I can’t imagine making quick changes to this without breaking a lot of stuff. It will take time to develop an intuition for predicting a change’s impact.

Now multiply my ramp up time by every dev on my team every time they need to make an update. You can imagine productivity slows down and cognitive overhead rises.

I agree that you don't want to spend time becoming super familiar with everything that you use but you should ALWAYS have a high level idea of what's happening under the hood. When you don't it inevitable leads to garbage in garbage out.

Way before the arrival of personal computers, it was clear for me to realize that I would be highly disappointed if average programmer abilities declined half as far as they have by now.

Some of the most coherent technical projects are the result of a single individual's outstanding vision.

Occasionally, the project can be brought to completion by that one individual, in ways that simply could not be exceeded alternatively, all the way from from fundamental algorithms through coding to UX and design.

Additional engineers could be considered useful or essential simply to accelerate the completion or launch.

When the second engineer is brought on board, it may be equally performant to have them come fully up to speed on everything the first engineer is doing, compared to having them concentrate on disparate but complementary needs which need to be brought to the table in addition.

If they both remain fully versed, then either one can do anything that comes up. Sometimes they can work on the same thing together when needed, other times doing completely different things to make progress in two areas simultaneously.

You're going to end up with the same product either way, the second engineer just allows an earlier launch date. But truly never within half the calendar time, those man-months are legendarily mythical.

For projects beyond a certain size more engineers are simply essential.

Then ask yourself what kind of team would you rather have?

All engineers who could contribute to any part of the whole enchilada from primordial logic through unexceeded UI/UX?

Or at the opposite end of the spectrum, all engineers who are so specialized that the majority of them have no involvement whatsoever with things like UI/UX?

Assuming you're delivering as satisfactory a product as can be done by the same number of engineers in the same time frame.

You're NOT going to end up with the same product either way.

Now aren't you glad it's a spectrum so it gives you an infinitely greater number of choices other than the optimum?

As long as the number of excellent programmers is steady or increasing, I'm not too disappointed if there's a bunch of average ones too, as long as those average ones have great tools that let them still make good software.

It does seem like some of the very top innovative projects are done by one person.

I have very little direct experience with software small enough for an individual to understand, but it seems like a lot of our modern mega-apps are elaborations on one really great coder's innovation from the 80s, and real knock your socks off innovation only happens every few years.

Engineers that don't understand or care about UX can be a really bad problem, attempting to bolt on a UI on something meant to be a command line suite is usually highly leaky.

The opposite seems to be slightly less of a problem, to a point, almost nobody writes sorting algorithms, and writing your own database is usually just needless incompatibility.

I definitely am glad it's a spectrum, because having absolutely zero idea about the real context gets you in trouble, and stuff like media codecs are hard enough we'd lose half the worlds devs(including me) if you needed to understand them to use them.

Writing performant code unfortunately tends to require at least having a basic working model of how all your dependencies work. Otherwise, you tend to find yourself debugging why you ran out of file handles in production.

I'm referring more to things like FOSS JS frameworks with the kind of batteries included design that prioritizes abstraction and encapsulation over simplicity. Nothing actually stops you from learning them, it just takes time, because their big, and it's not necessary to use them.