Even more, it's probably better to start off with a full understanding of these systems, so you can recognize when you're Greenspunning one of these abstractions, and just switch to using a robust off-the-shelf implementation of the same abstraction. (E.g. switch from writing a hand-rolled parser, to using a real parser-combinator library, or writing a formal grammar for a PEG compiler.)

Emphasis on "off-the-shelf": the most important thing to get from experience with these abstractions, is the understanding that you it will be less work to use the abstraction that has more power; and, in fact, that much of the tooling around these abstractions is built to assume that you want the full power of the tool by default, such that you have to add configuration to get less. So the workflow is actually "start off ordering 'one with everything', and then streamline your system over time, removing bits and pieces, as you notice properties of your particular situation that mean that you can do with something less powerful at particular stages."

eg: log4j -- so much power you can enable random bitcoin miners with your infrastructure (tongue in cheek)

ie: Sometimes it's not about more power but about the smallest/least powerful implementation that accomplishes the job.

I agree that using off-the-shelf is often important but it also comes with the need to vet the thing in the first place. That's not always trivial if you care about packaged licenses from dependencies or a lack of hidden features or actively developed or a plethora of other dimensions to what the "shelf" provides.

I think the difference is that something like log4j has no pressure to be "the least power it can while doing the biggest job it can" — it has mostly enterprise customers who want the power, but don't care about the scope creep. Something like LLVM, on the other hand, has both big enterprise use-cases, and tiny embedded use-cases, with big players invested in both. So its size and scope have been much more well-considered, with any components that do something other than "compile these data structures representing inputs, into these other data structures representing outputs" having been long factored out into additional "plugin" sort of downstream-dependency libs.

IME the best compromise I've found is to learn enough to recognize the problems, so that when I implement a bastardized compiler, I'm aware that that's what I'm doing and that at some point in the future I'll need to decide how deep the implementation should go or switch to actually using some library.

At some point, it's all just Computer Science. And not too much of it, either! An amount that could easily be learned in a (good) 4-year undergraduate program.

> What is a SQL query planner but a compiler?

Kind of? Lexing, sure, and to some extent IR, but an execution plan is not built from an instruction set, especially when you start talking distributed. Even if you ignore the distributed scenario, there's still a huge difference between understanding indices/columnar-storage/row-storage and implementing PGO or LTO.

Plus, there's also a lot of domain complexity here- are you aiming for just ANSI compat, or are you going deeper and trying to be compatible with specific SQL Server / Oracle / Postgres features?

> Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator?

Knowing little about HPC or GPUs, all I can say is that yes, there's absolutely a difference between the kinds of issues a kernel, hypervisor, and distributed scheduler have to deal with. Hypervisors need to integrate with higher-level RBACs, for one; distributed schedulers also need to be able to evict bad machines from the same system.

Telemetry also changes massively: if you're dealing with a kernel/hypervisor, stats from things like strace etc are useful. If you're distributed? You now have multiple cardinality problems (hostname, network devices, microservice release ID).

> Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store?

Again, yes (at least I assume so, I'm not super sure exactly what "NVMe storage pool" is, I assume you're talking about the protocol that SSDs speak) - e.g. what's your resilience story against a network failure (someone cutting a fiber line, or losing a subgraph of a clos fabric)?

IMHO you can do slightly better — you can know enough about isomorphic decisions you've made in other domains to drive design in a new domain, without necessarily understanding implementation.

(Especially "design" in the sense of "where does the shear-layer exist where I would expect to find libraries implementing the logic below that layer" — allowing you to understand that there should exist a library to solve constrained problem X, and so you should go looking for such a library as a starting point to solving the problem; rather than solving half of problem X, and then looking for a library to solve overly-constrained sub-problem Y.)

Or, to put it another way: knowledge of strategy is portable, even if knowledge of tactics is not. An Army General would — in terms of pure skill — pick up the job of being a Naval Admiral pretty quickly, because there's a lot about high-level strategy that isn't domain specific. And that experience — combined with hard-won domain experience in other domains — could guide them on absorbing the new domain's lower-level tactics very quickly.

Or, another analogy: linguists learn languages more easily than non-linguists, because 1. it turns out that, past a certain point, languages are more similar than they are different; but more importantly, 2. there are portable high-level abstractions that you begin to pick up from experience that can accelerate your understanding of a new language, but which you can also be taught academically, skipping the need to derive those concepts yourself. (You still need practical domain knowledge in at least some domains to root those concepts to, but you can do that in your own head when studying the concepts in a book, rather than "in anger" in the field.)

This belief — that there exists portable high-level strategic knowledge to software engineering — is exactly why some companies promote programmers into product managers. A product manager will be tasked to drive the strategy for writing a new piece of software, potentially in domains they've never worked in. They won't be implementing this software themselves — so they don't actually need to understand the domain with high-enough precision to implement it. But they'll nevertheless be relied upon to make architectural decisions, buy-vs-build decisions, etc. — precisely because these decisions can be made using the portable high-level strategic knowledge of software engineering.

(Though, once again, this knowledge must be rooted in something; which is why I don't really believe in CS / SWEng as programs people should be taking fresh out of university. The best time to learn them is when you already have 10 years' programming experience under your belt. They should be treated as "officer's finishing school", to turn an experienced apprentice 'programmer' into a professional Software Engineer.)

On to specific hand-wavy rebuttals, since my original rhetorical questioning didn't do the job of being particularly clear what I meant:

> but an execution plan is not built from an instruction set

What is a logical-level write-ahead log, but a long bytecode program that replicas execute? What does executing SQL do, but to compile your intent into a sequence of atomic+linearized commands — an instruction stream — in said write-ahead log?

(Alright, in a multi-master scenario this isn't as true. But you can get back to this property if your DBMS is using a CRDT event-streaming model; and AFAIK all the scalable multi-master DBMSes do something approximating that, or a limited subset of that.)

> Hypervisors need to integrate with higher-level RBACs, for one; distributed schedulers also need to be able to evict bad machines from the same system.

Perhaps you haven't dealt with machines of sufficient scale—the abstractions really do come back together! The IBM z/OS kernel knows about draining+evicting hotplug NUMA nodes as schedulable resources (and doing health-checks to automatically enable that), in exactly the same way that Mosix or TORQUE or kube-scheduler knows about draining+evicting machines as schedulable resources (and doing health-checks to automatically enable that.)

Even regular microcomputer OS kernels have some level of support for this, too — though you might be surprised why it's there. It doesn't exist for the sake of CPU hotplug+healthcheck; but rather for the sake of power efficiency. An overheating CPU core is, at least temporarily, a "bad" CPU core, and on CPU paradigms like big.LITTLE, the entire "big" core-complex might be drained+evicted as a response! That code, although different in purpose, shares a lot algorithmically with what the distributed schedulers are doing — to the point that one could easily be repurposed to drive the other. (IIRC, the Xeon Phi did exactly this.)

> e.g. what's your resilience story against a network failure (someone cutting a fiber line, or losing a subgraph of a clos fabric)?

My point with mentioning hierarchical persistence+caching for a distributed key-value store, is that these same problems also occur there. From the perspective of a client downstream of these systems, a network partition in tiered storage, is a network partition in tiered storage, and you have to make the same decisions — often involving (differently-tuned implementations of) the same algorithms! — in the solution domain for both; and, perhaps more importantly, expose the same information to the client in the solution domain for both, to allow the client to make decisions (where needing to capture and make this information available will heavily constrain your design in both cases, forcing some of the similarity.)

That's certainly true!

I'm not at 5 years yet in the industry myself, but I've definitely started to notice these patterns - the strategy vs tactics is a nice way to describe it :)

> Perhaps you haven't dealt with machines of sufficient scale—the abstractions really do come back together! The IBM z/OS kernel knows about draining+evicting hotplug NUMA nodes as schedulable resources (and doing health-checks to automatically enable that), in exactly the same way that Mosix or TORQUE or kube-scheduler knows about draining+evicting machines as schedulable resources (and doing health-checks to automatically enable that.)

Oh, this is interesting. My experience has been as a Google-internal user of Borg, where although the fleet is large, most of the machines are individually small enough that it's easier for us to simply evict a machine from the fleet rather than block a core. (The whole "commodity" server strategy, whereas z/OS follows a very different architecture, but I see exactly what you mean there.)

i think any kind of undergrad program can only ever scratch the surface of some of these topics. people spend decades becoming proficient in just a single vertical (microchip architecture, distributed systems, asics/fpgas, signal processing, gpu shaders, network engineering, jit compilers, language design, color spaces, database engines, etc).

sure, it's all "just" CS/EE :D

look at how much knowledge/skill was required to make an even faster fizbuzz: https://news.ycombinator.com/item?id=29031488

The Programming Languages course I took in college had us implement a toy subset of Ruby targeting the Lua VM. Easily the most interesting and useful class that I had the pleasure of experiencing; it completely changed how I think about languages. I think everybody should have exposure to the basic principles of compilers like syntax trees, flow analysis, calling conventions, and so forth (if not necessarily the more complicated stuff like operational semantics).

Nice unintentional pun.

1. They are one of the most researched areas of CS with a large amount of educational materials, tools, and community knowledge to help.

2. They map to any problem involving transforming an input into an output (hell, any problem transforming an input to an output is a compiler or interpreter).

knowing how to think about a program interpretation space gives quite a broader view IMO

It sounds really basic, but the idea of being handed a description, and doing/creating something based on that is everywhere. It is really quite beautiful.

In a sense, compilers literally build a bunch of linguistic 'worlds' within their pipeline, each one emphasizing something different.

> not sure that there is a difference to compiler as stated in the article

It's hugely different.

If you're building a compiler for an existing language, you inadvertently signed yourself up for implementing and supporting the entire language, which is a pretty large engineering task.

But if you design a new language, you signed up for not just implementing the language, but also:

- Actually designing it, which is a very difficult task with hard trade-offs all over the place.

- Documenting it so that others can actually learn it. Technical writing is another hard task that takes years to master.

- Syntax highlighting and editor support so that it's as easy to work with as other languages. You will discover that users in the wild apparently use thousands of different editors, all of which you are expected to support.

- Static analysis tooling so that code navigation, automated refactoring, linting, etc. all work. Users today expect a rich IDE experience for every language they use. Also, you'll discover the hard way that the architecture of an interactive static analyzer is very different from your batch mode compiler, so now you're likely writing two front ends.

- Thousands of answers on StackOverflow, blog posts, introductory videos, etc. Users learn in many different ways and they expect to find education material in the form they prefer.

So for example when you want to create a validation system for something, and then you realize that you want to have generics/type-variables/polymorphism, you want validation containers. Bam, the complexity suddenly shoots up! Or maybe you just want to let someone configure with yaml, and then you find yourself with substitutions and substitutions in substitutions and all that mess. Why?

The abstract reason is that the underlying mathematics of template substitution is lambda calculus, which is Turing complete. It's hard to learn lambda calculus because it's very abstract, it takes place in a mathematical ideal realm where nothing happens, but this also means that the same pattern reappears in a whole bunch of different contexts, and it is best if we just regard it as a design pattern and conscientiously adopt it. “Oh, this is a Lisp, I have seen the 20 line lisp-in-lisp evaluator, let's just code that in whatever host language and someone else will have thought of the details.”

Some of us might argue that you have to do that anyway, for any software you write. In fact, a similar argument might apply to your first point.

On the other hand, the last three points are really not requirements unless you are creating a general purpose language and have a large audience.

Yes, but if your software is another implementation of an existing language, the burden is an order of magnitude smaller because almost all of the existing docs for that language apply to your implementation too.

> On the other hand, the last three points are really not requirements unless you are creating a general purpose language and have a large audience.

Even if you have a small audience, if you want them to be productive, they need good tools and resources. Part of the reason why it's hard for DSLs to be effective is that most of the resources get amortized across all users. When you have a small audience, you can't justify building all these tools and docs, but the users still need them and their productivity suffers in the absence.

If your audience is too small, I think it often just doesn't make sense in terms of cost to make a language for them. You're usually better off using an existing language that already has those resources you can leverage.

Additionally, the dirty secret of all those things is they make orders of magnitude more sense to dig into and learn vs the innumerable arcane errors require Googling to fix (obscure Webpack errors, CSS hackery, etc). Your learning accumulates over time.

- a billing system

There's a reason why there are b2b companies that specialize in billing. When people branch out it's often billing and X, not X and billing. And even then sometimes they pull back. ADP bought a bunch of companies outside their wheelhouse and then spun them back out.

I can't vouch for the quality of what I've built, but taking time to attempt to build those kinds of things has always paid off in terms of overall betterment as a programmer.

Then I can go back to a day job of mundanely moving protobufs from one place to another.

I think the main thing is having respect for the mountain of engineering that went into existing solutions and not rolling your own unless you really really have to.

I think an Embedded DSL with post compilation constraint solver based analyzers could take the place of restricted mini-languages while taking advantage of the existing engineering and tooling available for modern languages.

This restricted subset concept has the huge advantage that you can slip out of the harness in a pinch. So you can reuse a relatively developed Float-less Python program in Full Python and later walk back removing floats.

That's really clever. It essentially obviates the whole configuration DSL tango; it also gets people started where the rubber hits the road -- not with syntax and parsing, but with why the DSL exists and what it's supposed to do and not do.

When it is released and as it matures we can extend it to do a bunch of other cool things like configuration DSL which would really help clean up Linux. Maybe even build scripts.

You could do stuff like this before with LISP, or Scala compiler plugins. But the tooling and accessibility will make a big difference.

Sometimes you start off with this cool idea/hack to solve a problem that could have been solved some other way. But your cool hack solves the problem and more! Then your fellow team members all want it to be even more powerful, and your cool hack slowly grows into a monstrosity.

In these situations, often your manager did not particularly want you to do this, but he let you do what you want. He does want you to be productive, and this doesn't count for that much. But since the team is now dependent on you, he does expect you to maintain it. You now end up with two jobs: The one your manager hired you for, and maintaining this thing.

I would love to write my own DSL to solve a problem at work, and this post read to me as a cautionary tale on pursuing that goal - unless management is aligned with your vision. If they're not, solve it the boring (and perhaps nonideal) way.

Unless when it somehow becomes the absolute necessity...or it's for your dissertation.

When you hit:

- a ERP

You start to build all the others!

(This is where I'm now!)

* A proprietary language

* A compiler for said proprietary language

* A way to define tables and views and turn these definitions into real tables and views that live in an SQL database. There is also a custom SQL dialect that gets translated to real SQL on the fly. I'm counting this as having its own database even though it offloads the actual database work to a real database.

* One of the products in this ERP suite includes a CMS built on all of the above.

Then what you have in your hands is a legacy monolith neglected over its life which might double as core of your stack and it's awful to move away from down the line.

Been there, don't recommend it.

Unfortunately, that's rather true of all software.

Anedoctal experience: a company I used to work for had all the core stack developed inhouse: programming language (yeah), compiler, database, etc. No one worked on it apart from the founder and a couple of other engineer that were no longer there.

A few years down the line a big client asked if we were able to integrate our software with their PLC and this had to be developed from scratch, as it wasn't in the scope of the programming language. The quote sent to them was in the six figures and up to 2 years for it to be ready. Every major programming language has a library that does this. Eventually the company lost the client.

At least with a DSL, you can often define a small core language and de-sugar down to that.

...and the alternate course of action is clear: use the tools you already have.

That's also where SAP's moat lies. Not with it's technological underpinning, but with the all the different industries and their processes which they've transferred into lines of code (and the ability to extend them with further LOC with the help of a "SAP consultant").

That’s not true for everyone. I’ve been a dev for 20 years and have never done any of those, and I have no desire to at all. That is ok! Programming isn’t some kind of progression to run from “n00b hello world” to “1337 compiler hax0r”, it’s a tool to solve problems with. Many of us will never have to solve these problems or have no interest in these problems.

So, if you’re like me, don’t worry because you’ve never made a brainfuck interpreter or a database engine. It doesn’t mean you’re a bad dev. You can have a successful career without building these! And if you need them some time, learn them at that time… you may not ever need to.

It's much wiser and far easier (assuming your goal is to have a successful career and run a profitable business) if you Keep It Simple, Stupid! Building a compiler/database/CMS/ERP instead of using an off-the-shelf one is, for well-selected business problems, a bad decision, increasing complexity, adding risk, and allowing scope creep.

But sometimes, in spite of your efforts to adhere to the KISS principle, the real world is hairy and some complicated new problems need complicated innovative solutions, and instead of turning a crank to glue together some off-the-shelf libraries, you'll find yourself, in my particular example, way off in the deep end getting PCBs manufactured or, in the article, writing your own compiler. It's not bad, it just means you've got a niche problem to solve! Or that sales did a terrible job of understanding what they were offering, and something basically equivalent to the customer is just an off-the-shelf Wordpress plugin, but instead you're stuck writing a custom CMS.

The trick is to recognize that point before you have a giant, unmanageable ball of ....

Maybe it's better put as "But sometimes all your product requirements strongly suggest that you have to build X and there isn't really a way to avoid it" (and what's available off-the-shelf doesn't fit)?

I mean, even fully caffeinated, I still manage to beclown myself on a semi-regular basis, but the 'semi' part is still a net win.