I guess you're using the word genius ironically, but I've also seen such things be radical in the sense that they're built in a way that no one else would have done, yet it ends up working perfectly for the business and being far more successful than anyone would have predicted. There's a certain type of mind that is immune to fashionable or normal technical choices, that often goes along with remarkable productivity. I suppose the word 'genius' might be a good fit for that.
The interesting thing is that such cases can also actually be the huge nightmare you write about, in the sense that there are tradeoffs in the original design that originally made sense but become harder to deal with later, but also in the sense that their nonstandardness can be hard for more typical engineers to work with. (This is often referred to as the 'hiring problem', though that can be misleading.) The trouble with the typicalness of more typical engineers is that they're steeped in a technical culture which they regard as the proper and obvious way to do things, but which really is mostly just the set of conventions they're used to. Those two things are easy to conflate when all your peers agree on what's proper and obvious (for example that you should never build a complex system entirely in Postgres stored procedures, or whatever the weird thing is). Those conventional assumptions are at least part of what makes such systems a 'nightmare', because if you're working on a system that feels 'wrong', then you fight it rather than blend with it, and that makes it harder to maintain. Everything that's objectively wrong with it will loom larger in your experience than it would if you didn't have the feeling that it should never have been built that way in the first place. Worse, many things that aren't objectively wrong with it, in the sense that they aren't intrinsic to the system or the domain, end up blowing up into major issues because people find they can't work with this system in the way that they're used to, and it's too hard to abandon ways that you're used to—it always feels like the system is at fault. Meanwhile, everything it got right will fade into the background as you assume that any system that had been built 'properly' (i.e. the way you'd have done it) would have gotten all those things right too, at the same cost or better.
It's hard for the later engineer who experiences the 'nightmare' to fully appreciate that if it weren't for that weird, obviously 'wrong' system, they wouldn't have their job in the first place (nightmare and all), because the business that succeeded was built on the weird system.
I don't know anything about this case other than what John posted above, but I've seen something like it in other situations, and in fact I would say that PG and YC have this in common with the case of Lee and Cloudflare as John describes it—as remote from one another as Postgres procedures and custom Lisps must be in every other respect.
What I wish we could teach younger engineers is: when you run across a system like this, pause. Then, pause. Then pause again. The creativity at the core of it can be remarkable. Often you see things that one person did which would have required a large team in other hands, and therefore would never have happened otherwise. Maybe that's one reason why these systems tend to be found at the start of companies: more conventional approaches wouldn't have worked at a stage when hiring a team wasn't an option yet; and on the other hand, the teams you do hire at the later stages tend to have more conventional engineers whose minds (or rather, whose conditioning) can't grok why you'd ever have built something so weirdly.
Usually the later engineers bond with each other around how much they dislike the 'legacy' system, and lobby to replace it, and usually get their way in the end. That's a cycle-of-life thing, but it's also sad when we prefer our conventions and assumptions over seeing, and honoring, the creativity in what came before. Sometimes masterpieces get paved over because of that. Of course, not every nightmare system is an unappreciated masterpiece. But some are, and the bias in our industry is extremely strong toward perceiving them as nightmares. I've seen people take chainsaws to masterpieces without realizing that that's what they were doing, and it makes me sad to think of it. But it makes me happy to read what John wrote to honor Lee, because that's how I'd want someone to write and feel about something I'd built, if I built something like that.
and in fact I would say that the case of PG and YC have this in common with the case of Lee and Cloudflare as John describes it—as remote from one another that Postgres procedures and custom Lisps may be in every other respect.
Oh, you reminded me of something fun. I had a lot of freedom under Lee when I first joined Cloudflare (that's partly how Cloudflare became a Go user) and one of my first jobs was writing a new WAF service... I made an entire working version in Arc. I almost made all our customers write rules using it.
And those conventions change over time. So you might write something using Java EJBs at some point and then the world moves on and engineers no longer know how those things work. What looked like a great architecture is now confusing for most engineers.
That's true, and weird unorthodox architectures can actually end up being more stable in the long run, especially if their design was an intrinsic fit for the domain. A weird architecture that is a good fit for its domain often has simple ways of meeting key requirements that would have been more costly to develop with more conventional techniques. I'm just hallucinating this from your comment upthread, but a requirement like "making sure audit logs are kept" is the sort of thing I mean. Any architecture can do that, but some make it much easier. When a business starts out with a weird architecture in its early days and ends up succeeding, it's possible that that weirdness gave some competitive advantage.
I think there is a lot of truth to this caricature, but it rubs me wrong the way you are painting the "later" programmers as blinded by fashion or tradition. "Geniuses vs ignorant sheeple" is a false dichotomy.
Yes, in some similar situations, some programmers will see things that way. But have some faith and respect for individuals' abilities to reason and apply their experience. As you implied, these kinds of systems are often optimized so that one person could run them.
So it's often a trade off — "this would not be easy for a medium or a large team to work on, but it's the easiest thing for me as an individual to maintain by myself that gets the job done." And that's OK. And yes, it is natural to shift over time. But this is precisely because you now want 40 developers maintaining it instead of 1.
Overengineering is real. But architectural tradeoffs that don't work well for large teams are real too.
> as remote from one another as Postgres procedures and custom Lisps must be in every other respect.
Just a fun fact: Parts of postgres (before the SQL days, I think) used to be in lisp. To this day we still have some weird function/macro names stemming from those (e.g. lcons).
* Once upon a time, parts of Postgres were written in Lisp and used real
* cons-cell lists for major data structures. When that code was rewritten
* in C, we initially had a faithful emulation of cons-cell lists, which
* unsurprisingly was a performance bottleneck. A couple of major rewrites
* later, these data structures are actually simple expansible arrays;
* but the "List" name and a lot of the notation survives.
Thanks for the touchingly poignant homage to creative problem solving. Even if we appreciate the effectiveness of such a solution, the sheer artistry in it is highly under-appreciated. Most people sadly walk around with so many blinders that they cannot really _see_. _"[...] pause. Then, pause. Then pause again."_ indeed.
I actually think that doing things in stored procedures could be a great hack to squeeze out some extra performance, and I'm sure that a system like this can accomplish some important business goals.
However, in the example I'm thinking of, it was much more dysfunctional. The advantage of stored procedures there was that the organization's release process was so broken that it was much faster for sales and marketing to get features added by going through Carl, the DBA, instead of submitting the request to the dev team. This was because Carl was able to add the logic by sending one query rather than going through the broken months-long release process.
Carl was actually a real nice guy, although hard to get a meeting with due to his great importance and busy schedule.
Reading code is harder than writing and separating taste from business logic is even harder. Often what looks like an ugly wart was a very compact representation of domain knowlege for a special use case. Untangling that knowlege and re-mastering that use case in a refactoring or even rewrite goes wrong far too often.
The interesting thing is that such cases can also actually be the huge nightmare you write about, in the sense that there are tradeoffs in the original design that originally made sense but become harder to deal with later, but also in the sense that their nonstandardness can be hard for more typical engineers to work with. (This is often referred to as the 'hiring problem', though that can be misleading.) The trouble with the typicalness of more typical engineers is that they're steeped in a technical culture which they regard as the proper and obvious way to do things, but which really is mostly just the set of conventions they're used to. Those two things are easy to conflate when all your peers agree on what's proper and obvious (for example that you should never build a complex system entirely in Postgres stored procedures, or whatever the weird thing is). Those conventional assumptions are at least part of what makes such systems a 'nightmare', because if you're working on a system that feels 'wrong', then you fight it rather than blend with it, and that makes it harder to maintain. Everything that's objectively wrong with it will loom larger in your experience than it would if you didn't have the feeling that it should never have been built that way in the first place. Worse, many things that aren't objectively wrong with it, in the sense that they aren't intrinsic to the system or the domain, end up blowing up into major issues because people find they can't work with this system in the way that they're used to, and it's too hard to abandon ways that you're used to—it always feels like the system is at fault. Meanwhile, everything it got right will fade into the background as you assume that any system that had been built 'properly' (i.e. the way you'd have done it) would have gotten all those things right too, at the same cost or better.
It's hard for the later engineer who experiences the 'nightmare' to fully appreciate that if it weren't for that weird, obviously 'wrong' system, they wouldn't have their job in the first place (nightmare and all), because the business that succeeded was built on the weird system.
I don't know anything about this case other than what John posted above, but I've seen something like it in other situations, and in fact I would say that PG and YC have this in common with the case of Lee and Cloudflare as John describes it—as remote from one another as Postgres procedures and custom Lisps must be in every other respect.
What I wish we could teach younger engineers is: when you run across a system like this, pause. Then, pause. Then pause again. The creativity at the core of it can be remarkable. Often you see things that one person did which would have required a large team in other hands, and therefore would never have happened otherwise. Maybe that's one reason why these systems tend to be found at the start of companies: more conventional approaches wouldn't have worked at a stage when hiring a team wasn't an option yet; and on the other hand, the teams you do hire at the later stages tend to have more conventional engineers whose minds (or rather, whose conditioning) can't grok why you'd ever have built something so weirdly.
Usually the later engineers bond with each other around how much they dislike the 'legacy' system, and lobby to replace it, and usually get their way in the end. That's a cycle-of-life thing, but it's also sad when we prefer our conventions and assumptions over seeing, and honoring, the creativity in what came before. Sometimes masterpieces get paved over because of that. Of course, not every nightmare system is an unappreciated masterpiece. But some are, and the bias in our industry is extremely strong toward perceiving them as nightmares. I've seen people take chainsaws to masterpieces without realizing that that's what they were doing, and it makes me sad to think of it. But it makes me happy to read what John wrote to honor Lee, because that's how I'd want someone to write and feel about something I'd built, if I built something like that.