> I question the need for scale in 90% of the places where the tech industry has cargo-culted it.
Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity.
> Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context.
Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it.
Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost.
Reimplementing and forking are generally not value adds. Building software is today about creating/adding value.
Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years.
> I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context.
I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.
> Reimplementing and forking are generally not value adds. Building software is today about creating/adding value.
Check out the writings of Carlota Perez[1]. She distinguishes between two phases in adoption of a technology: installation and deployment. Analogously, reimplementing adds value because you learn to make something better or faster or simpler. And you learn how to make it and can now take that knowledge elsewhere and use it in unanticipated ways.
It blew my mind a few years ago to learn that the Romans had knights: https://en.wikipedia.org/wiki/Cataphract. WTF, I thought they were a medieval thing. It turns out everyone in the Ancient world understood the idea that an armored person on a horse can outfight someone on foot. What was missing in Ancient Rome was the ability to teach people at scale how to fight on horseback. As a result, most cataphracts would dismount before battle. It took literally a thousand years for society to figure out how to educate people at scale on mounted combat. You don't really know how to do something just by building it once, and you don't unlock all its economic potential just by building it once. Societies have to build it over and over again, thousands of times, to really understand its value. And they gain benefits throughout the process.
> Analogously, reimplementing adds value because you learn to make something better or faster or simpler. And you learn how to make it and can now take that knowledge elsewhere and use it in unanticipated ways.
You know, there’s some truth in this. At the same time, there is only truth to the extent that the person reimplementing is actually striving to do better. I would bet that well over 99% of reimplementations are strictly worse than the thing they are mimicking.
I spent a bunch of time implementing a custom windows framework similar to WTL. It was a great learning experience. I would also say that in no way woks it have been useful to anyone for or to appear in an actual product. In most ways it was doubtless inferior to WTL. I used it for toy projects and that’s what it was.
> If I write some code, I’m realistically the expert.
Ha, disagree. This is overly general and reductionist.
See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.
It's bad now. It might not have been bad when it was originally written. Back when Unix was first being developed, you had machines with 64K or 128K of RAM, which is not much, and because of that, software was much simpler, and such design compromises could be kept in mind. And when C was standardized in 1989, it might have been a bad idea at that time, but the standards committee didn't want to obsolete all the existing code (a major concern at the time). But by all means, get a time machine and plead your case.
As an example of coding for size, I recently wrote a copy of strcpy() with similar semantics. I had to, because of constraints on memory. I wrote an MC6809 (an 8-bit CPU) disassembler in MC6809 assembly code, and the prime consideration was code size. Strings ending with a NUL byte? My GOD! that will waste 80 bytes! Waste a byte on size? That's another 80 (or 160 if you insist on 16 bits for size) bytes wasted! Instead, since all the text is ASCII, set the high bit on the last character. That more than makes up for the bit of oddness in my implementation of strcpy() to deal with the last byte, and helped contribute to the code being 2K in size.
strncpy() is for a different context, and it won't necessarily NUL terminate the destination [1]. It's not a compromise for strcpy(). strlcopy() will NUL terminate the destination, but it first appeared in 1998, nine years after C was standardized, so again, time machine etc. Should Ken Thompson & co. included strlcpy()? It's an argument one could make, but I'm sure they didn't see a need for it for what they were doing. Oversight? Not even thinking about it? Yes, but I don't blame them for failing to see the future. Should Intel not do speculative execution because it lead to Specter?
Now, assuming strlcpy() existed in the 70s---is that just as efficient as strcpy()? Should strcpy() have always included the size? Again, I think it comes down to the context of the times. Having worked on systems with 64K (or less! My first computer only had 16K of RAM), and with the experience of hindsight, the idea of implementing malloc() is overkill on such a system (and maybe even 128K systems), but I won't say people were stupid for doing so at the time.
[1] I think strcnpy() was there to handle the original Unix filesystem, where a directory was nothing more than a file of 16-byte entries---14 bytes for the filename, and two bytes for the indode.
I'm getting a little impatient with all this argument in circles. The point is not what someone should have done with hindsight. The point of this little sub-thread is that saying the author is always the expert is oxymoronic. Expertise is often multifaceted and distributed.
The larger point is also that it should be possible to get rid of things once hindsight has found them wanting.
> The point of this little sub-thread is that saying the author is always the expert is oxymoronic.
Writing the code makes me the expert in that code. Meaning how it works. Why it was built the way it was. How it was intended to be extended. Yeah, it might be shitty code and maybe someone with more domain or even just general expertise would be able to do better. But if I write the code, I’m still the expert in that specific code. Someone else can of course roll in and become and expert in that code, too. Expertise in my code is not a bounded resource.
If you took my comment to mean that writing the code makes me the expert on all possible implementations or choices for the solution, or that I am best qualified to decide what the best solution is, I must have communicated poorly.
(Also, this is pedantic, but that’s not what oxymoronic means.)
> If you took my comment to mean that writing the code makes me the expert on all possible implementations or choices for the solution, or that I am best qualified to decide what the best solution is, I must have communicated poorly.
No, I didn't think that. You're all good there.
> Writing the code makes me the expert in that code. Meaning how it works.
Go back and reread the bottom of https://news.ycombinator.com/item?id=30019146#30040731 that started this sub-thread. Does this definition of 'expert' really help in any way? Ok, so some author wrote some code, they're expert in that code. Big whoop. I don't want to use that code. I wrote a shitty simpler version of what you wrote, now I'm an expert, you're an expert. How is this helpful?
I think the only expertise that matters is domain expertise. Software should be trying to communicate domain expertise to anyone reading it. So that anyone who wants to can import it into their brain. On demand.
The “expert in the code” matters in terms of siloing. That was the context where I mentioned it. The engineers who work in an area all the time and know the code best are typically best equipped to make the next changes there. Hence a silo. Maybe I completely misunderstood your comment about late binding experts?
But stepping back for a moment, I’m increasingly confused about what your thesis here is. You’ve clarified that it’s not actually about abstraction. It’s also apparently not about clearly factoring code (though maybe you think that contributes to problems?). What is the actual thesis?
Engineers should be experts in the entire code base?
Code is in general too hard to understand?
Something else?
It feels like we’re discussing mostly tangents and I’ve missed what you intended to say. If I understood your message, I might have a better mental model for the utility of copying code.
Here's another attempt, a day and a half later. My thesis, if you will.
Before we invented software, the world has gradually refined the ability to depend on lots of people for services. When we buy a shoe we rely on maybe 10,000 distinct groups of people. When we write software we bring the same social model of organization, but all the vulnerabilities and data breaches we keep having show that it doesn't work. https://flak.tedunangst.com/post/features-are-faults-redux enumerates (in better words than mine) several examples where services made subtly different assumptions, and the disconnects make their combination insecure.
Since we don't know how to combine services as reliably in software as in non-software, we need to be much more selective in the services we depend on. A computer or app just can't depend on 10,000 distinct groups of people with any hope of being secure.
Yeah that's a failure on my part. Like I said, I've spent ten years trying to articulate it and spent a few hours in conversation with you, and I still can't seem to get my actual thesis across. I don't think another attempt right now will help where all else before it has failed :) How about we pause for a bit, and we can pick it up again in a few hours. Perhaps even over email or something, because this back and forth is likely not helping anyone else. My email is in my profile.
> Division of labor accumulates bad ideas along with good ones.
Agreed. To take an example from my area of specialization, accessibility, both Windows accessibility APIs (yes, as is often the case with Windows, there's more than one) have serious design flaws IMO. But they still allow disabled people to do many things we would otherwise be unable to do. I worry about what will happen if programmers truly follow your proposal from another subthread: "build/reuse less. Do without." In this case, doing without would make computers much less useful for many people, and possibly erect new barriers to employment, education, and just living one's life.
What you’re actually saying is that you have the benefit of hindsight. That in no way makes you more on an expert than Ken Thompson who also has the benefit of hindsight.
I would guess that literally thousands of people have implemented an equivalent to strcpy with exactly the same issues (and likely more). Reimplemention does not imply a better implementation.
Decades of experience using strcpy is what taught us it’s a bad idea, not reimplementing it a thousand times.
Maybe it took you decades. One decade sufficed for most of the world.
This is starting to feel like a pissing contest. Go write your software thinking you know more than your users. Me, I constantly need to talk to mine. They're often more expert on the domain than me.
> This is starting to feel like a pissing contest.
Is there some reason it suddenly went from us discussing a difference of opinion to hostility? I have no idea what this is about.
> Go write your software thinking you know more than your users.
That’s a strange nonsequitur. End customers don’t generally tell me how to craft software. When I work with customers, it’s to understand their domain and their problems, not mine.
Again, not sure where this hostility is coming from.
Perhaps I'm coming down with something. I'm uninterested in the sub-thread past https://news.ycombinator.com/item?id=30019146#30042174. It feels like willful misinterpretation, like you forgot what I said 2 exchanges ago when you respond to any single comment.
I've said my piece in a bunch of different ways, over ten years and again ten times to you. Perhaps this is as close as we're going to get to mutual understanding, if you think after all this that I'm trying to one-up Ken Thompson or something.
I think you're using words like "expert" and "strictly worse" way too bluntly. What's better for the author is often not what's better for the user.
Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity.
> Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context.
Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it.
Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost.
Reimplementing and forking are generally not value adds. Building software is today about creating/adding value.
Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years.
> I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context.
I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.