Regarding applications to modern software engineering, this parable is basically about operations surrounding architectures that promote long-lived processes and rare reboots (VAX) vs. ones that promote quickly-spawned, ephemeral processes and which rely on reboots for cleanup (Unix).
The modern comparison I would make is between the design of scale-neutral twelve-factor apps in language-platforms made for quickly spawning ephemeral processes ("scripting" languages like PHP, Ruby, Node, Lua, or even—in practice—Go)† and the design of "living system" VMs—and languages thereupon—that load and unload modules over time without shutting down (JVM app servers, Erlang, etc.)
It's okay to effectively "reboot" your Heroku instances to get them into a sane state. Not so with your databases, stateful load balancers, or VoIP servers. They come from two different worlds: one eschews state, while the other owns and manages it.
† Notice how all these languages are effectively wrappers around the same paradigm. There's a strong sense in which a "scripting" language is just a language that models the world in the Unixine fashion, such that it can be used to glue other Unixine programs together.
ones that promote quickly-spawned, ephemeral processes and which rely on reboots for cleanup (Unix).
The only modern OS I know of that "relies on reboots for cleanup" is, ironically enough, the one famously descended from VMS (I am of course talking about NT-kernel-family Windows).
And this is not a VAX vs. Unix story; it's a story about by-the-numbers IBMers who dress in the Right Clothes and say the Right Things being completely feckless in the face of a VAX, development and maintenance of which has quite a different tradition and culture, mainly consisting of scruffy guys who Get Shit Done. Of which there is not a lack in the Unix tradition either.
A parable communicates a lesson, and the lesson "don't let ops people who have never run an architecture resembling yours near your machines" is kind of vacuous/trivial. Like I said, I'm looking for a piece of wisdom can be extracted from this story that's relevant for modern devops people.
I think the lesson is basically: some systems evolved around doing discrete batch-processing tasks that throw away their arenas when they're done, and have very few and discrete connection points; while other systems are "about" living object graphs that can't be disentangled, only upgraded as an entire system, and which maintain themselves rather than having maintenance done upon them from outside.
Rebooting solves problems with the former, since you're cleaning up all your temp files, cached intermediary results, etc. and then scheduling fresh new tasks. Rebooting the latter just makes it struggle to put itself back together exactly the way it was before. Think of power-cycling an LAMP "app-tier" server (Unixine) vs. a database server (Vaxine.)
> In what case would it be that disastrous to reboot a database server?
To assure data integrity, an interactive database system has to log off all its users and shut down open databases before rebooting. If the reboot can't be orchestrated that way, if users can't be logged off in an orderly way, there will be data loss.
If your database is ACID, you can pull the power on the server from the wall and have it instantly turn off w/o data loss. That's the entire point of transactions...
This is the basic reason why raw devices where the way to go for decades for use as database devices and are still in use in most installations.
I'm talking about Sybase here. But any database, which shifts the commit responsible to the operating system (without explicit guarantees that committed data is indeed written to disk) may invite you in for a huge and very unpleasant surprise.
Counting on raw devices when you have a SAN underneath presenting LUNs to the server means unless you are very careful, it does not mean that said write necessarily made it all the way onto disk.
> If your database is ACID, you can pull the power on the server from the wall and have it instantly turn off w/o data loss.
Yes -- all except the ongoing transactions at the users' workstations. That by itself could represent a serious synchronization problem. For a serious business, an emergency power system might represent a better choice.
If you've not pulled the plug on your users' workstations, the clients should be perfectly capable of retrying the broken transactions when the database comes back up. There can, of course, be constraints that make this impossible, but sloppy coding making it impossible is more common.
UNIX isn't mentioned. When they say IBM I believe they are referring to System/360 and its decendents. This is a story of two mainframe competitors. It has nothing to do with UNIX.
In the context of the time, the VAX is the more interactive/flexible/etc. system relative to the 360. It certainly wasn't viewed as a mainframe then. However, from today's vantage point, VAXen--and even the large UNIX servers that followed--seem very mainframe-like compared to volume x86 servers.
It's about IBM mainframes (not Unix systems) on an uninterruptable ("immortal") power supply, a VAX which wasn't, but which could survive reboots in pretty good shape. Except when handling world-changing financial transactions.
Where did you get the idea that the story had anything to do with Unix? Or that Unix relies on reboots for cleanups?
Note that around the time of this story, one of the most popular hardware platforms for running Unix (especially in academic environments) was ... the VAX (and its predecessor, the PDP-11). So in fact of the 2 "sides" in the story (IBM mainframes vs. VAXen running VMS), Unix would be on the scruffy VAX side (if anything, at that time Unix folks were scruffier than VMS folks).
Is there any support at all for the implication that this anecdote has any more than a humorous and coincidental relationship with the 1987 stock market crash?
I've run across this story more than once. It's amusing but also more than a little troubling if that is in fact how fragile the global financial system is.
Related are the faster-than-light stock-exchange trading orders of the past few years.
Some trades were originating out of Chicago for NY exchanges which, based on release timing of information, would have had to travel faster than light. Truth being that there's early access being offered some traders.
There are huge disasters that come of software mistakes -- but look at the story of the Therac-25 vs. this kind of story. Factual stories don't normally have punchlines.
If however many billions of dollars were actually lost due to a server rebooting at the wrong time, would it actually be possible that it remained a secret, only documented in this one humorous and totally un-sourced account?
Also note the exaggerations of the incompetence of the IBM datacenter operators.
It's an entertaining read, because it's written to be an entertaining read ("a former gnome of Zurich"?). Maybe there's some kernel of truth in there; maybe not -- but there's no way to tell now (without other, better sources) what that may be.
The modern comparison I would make is between the design of scale-neutral twelve-factor apps in language-platforms made for quickly spawning ephemeral processes ("scripting" languages like PHP, Ruby, Node, Lua, or even—in practice—Go)† and the design of "living system" VMs—and languages thereupon—that load and unload modules over time without shutting down (JVM app servers, Erlang, etc.)
It's okay to effectively "reboot" your Heroku instances to get them into a sane state. Not so with your databases, stateful load balancers, or VoIP servers. They come from two different worlds: one eschews state, while the other owns and manages it.
† Notice how all these languages are effectively wrappers around the same paradigm. There's a strong sense in which a "scripting" language is just a language that models the world in the Unixine fashion, such that it can be used to glue other Unixine programs together.