The author's key takeaway is "flow is king", but what stuck me most was giving people freedom to do their job -- e.g., giving the nurses unrestricted access to the medication, letting the CT tech focus on their job, roaming ER doctors, delegating triage to a nurse, etc. -- the success here came from giving people responsibility and trusting them.
I think a key here is recognizing when the expected result is catastrophe and acting accordingly.
There's a whole lot of procedures in the hospital that are designed for normal times and are _good_ in normal times(*) - they enforce people double-checking their results, they give radiologists time to make the right call, they make sure the right drugs are going into the right people, they make sure patients are prepped correctly for a procedure. All of that takes time, but it moves your expected outcomes from, say, 95% to 99%.
In the case described, the expected outcome is 0% - it's a fucking catastrophe, there's 250 people on their way, and everyone who comes in is a corpse waiting to happen. At that point, absolutely, flow is king - hitting an expected outcome of even 50% (looks like they got somewhere around 65%) is a massive upgrade, so yeah, shove as many people through the CT scan bay as you can, let your radiologist flip a coin, and if the nurse gives the wrong drug, fuck it, that's probably not what killed the patient - move on and hope you do better next time.
I think the real takeaway here is recognizing what the stakes of your situation are and acting accordingly - what are the outcomes that matter, and what's the set of activities that get you closer to your desired ones. Sometimes that's care and caution, and sometimes that's taking your best guess, committing to it, and accepting the outcomes.
Huge kudos to the author and to whoever set up the system that enabled them to act as needed and supported them in doing so, though - I'm sure they broke every regulation in the book over the course of those six hours. Plenty of other disasters have been made much worse by the lack of flexibility in a system to recognize extreme circumstances and act accordingly.
(* take "good" with all the caveats you want here. I get it.)
It’s not so much freedom to do their job as getting them out of the trained pathways that work well in the nominal case and _then_ letting them use their judgement.
Requiring people to double check that someone gets the right meds is super important 99% of the time. Having a radiologist review x rays asynchronously is more throughput efficient 90% of the time. Having the person with the most experience do triage is normally the best use of his time and saves lives.
All of these are based on assumptions that cease to be true in a mass casualty event.
An irritatingly overlooked issue is the tension between procedure and results. Obviously, a well-refined procedure learns from past issues and evolves, growing ever more adaptable and useful, the way that aircraft rules are "writ in blood." Still, every one of the adaptations was prompted by an event that was Not in the Script.
I think, for my next job, one of those questions I will ask when the time comes to ask if I have anything I am curious about in the interview is the choice, when a choice must be made, between procedure and results ... which am I expected to prioritize?
There’s not really a clear tension though. Procedures exist with an intent to get results. Airlines have lots of procedures that are carefully followed and the result is “the plane always arrives safely”. If you asked a pilot the question to “make a choice between procedures and results” they’d think you were crazy.
Obviously there can be misguided procedures, but any decision made or operation done can be misguided.
Again, and continuing with the flight safety, all of it was an iterative process, learned the hard way. Flight safety rules did not emerge whole and unchanged with the Wright brothers. Every change was prompted by one or more incidents during which the current procedure was not appropriate.
What then?
Consider the current thread on the whole "toaster in the dishwasher" topic, during which someone related an incident wherein an entire server site was immersed in water but still functioning (https://news.ycombinator.com/item?id=41251234). The site manager followed procedure (wait a while, not cut the power, perform risk assessment) and it resulted in total loss, but the poster wanted to "cut the power, pump the water out of the bunker ASAP and immediately clean the whole lot with pure water." Here we have a tension between procedure and results. Procedure ended up causing total site loss, which was completely avoidable.
Similarly, a current thread on an ER doctor not following the usual procedures during a mass casualty event was lauded. A choice had to be made. Here, results won.
I just like to know this sort of thing about a work culture in advance. Letter of the law versus the spirit of the law, and so on.
How would you know? How would they know? Okay, sure you will have a hunch based on the answer, but it seems something that only experience can tell. (Okay, you can ask them about their experience.. and who know how forthcoming and honest and relevant their answer will be, right?)
Pilots aren’t limited by experience, they know a great deal about aircraft systems and the mechanics of flight. Procedure is incapable of handling every single edge case because the possibilities are endless.
So when do you deviate from procedure? When the edge case you’re in is well outside the scope of exit procedures which requires detailed understanding of the procedures and their justifications.
Pilots are also allowed to deviate from any procedure and even violate federal regs in the event of an emergency.
14 CFR § 91.3(b) In an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule of this part to the extent required to meet that emergency.
What's the result of the completely useless flight safety mantra? People will not leave their belongings! The mantra should tell them that they will be compensated if they make it out alive but they will be put down like a mad dog if they hinder others with their packing and packages...
Also people do charge all the fucking time, and don't put their phone into airplane mode. :|
I would say it works incredibly well. Everyone knows what it is. Everyone knows what they are supposed to do in an emergency. Everyone knows there are emergency exits. Everyone knows there are life jackets. Everyone knows they are supposed to wear a seatbelt. At least some people will get it right.
Do not underestimate how even minimal training can be extremely effective at scale. I know from experience that I am a person who does not freeze, I focus. When shit happens, I act but I can act without thinking. Because of that mantra, I have a plan to follow and I will act correctly.
Also don't underestimate the effect of priming. It reminds you there is a plan. When pilots prepare for takeoff, they briefly review how to handle emergencies during takeoff. Not because they don't know, but to bring those procedures to attention and have them ready in their mind.
For the mantra to work, all it has to do is remind you those procedures you vaguely know exist.
I believe this is not a question that can be asked.
Recently I pushed a change straight to production by myself with no approval and violated many rules but it saved us and carried us for weeks. Worse case if it broke, rollback would happen under in under 30 seconds.
I did it because not only did I triple check, I’ve kept mental track of the number of regressions and issues that have been logged against all of my work throughout my career. I’m good at determining risk and I know my bug rate is very low (I git blame every bug to find out who and why caused it. I don’t tell my coworkers but it does play a role in who gets what kind of tickets.) I did what I did because frankly I know it was going to work and no one was going to complain. And truthfully, I’ve done this at different companies several times. Of course, it’s still never a light decision and I rarely ever do it.
But if someone asks me if they could do the same thing, I would not be able to tell them. They would have to keep track of the same details and to be honest, if someone is asking if they can break rules, they probably shouldn’t.
I once had an engineer push changes directly to production because they were confident that they would do it correctly.
In fact, they were completely misguided about their own accuracy because they had systematically ignored or not understood the errors that they had made in the past.
The challenge as a tech lead or manager is telling the difference between you and them. Or even telling whether you are them. My own tendency would be to fire both of you.
What struck me was how the author was able to identify choke points and make decisions to deviate from procedure to fix them. In many organizations, adherence to procedure is more important than getting good results. It is interesting to see that at least in this hospital at this time, that wasn't true.
He mentions that the CEO of his hospital was there and accepted his suggestion to reorganize the CT scan flow. That shows a level of trust and flexibility that's not common in large organizations.
This struck me too—especially the way he seems to have kept one eye on the big picture even while he was facing a flood of individual patients depending urgently on his task-level attention.
Having people bleed out in the waiting room is quite an unpopular outcome. When you have the luxury of time it's less excusable to make a misdiagnosis or to rush a treatment.
While that’s true, I worry about glorifying too much the desperation of the whole situation.
If the same patients had arrived at a pace consistent with normal operations, wouldn’t you expect the outcomes to have been better?
Normally cross-checking medications and dosages saves lives by reducing medical errors. Normally I’d rather not be operated on by a surgeon so strung out that, like the author describes, his mind can’t make sense of words on a page.
There are times when the problem vastly outmatches the resources you have to bring to bear. Where getting things done at all is more important than making sure they’re done absolutely perfectly. But to me the lesson is in deciding which compromises will save the most lives.
Which has to weigh even heavier on the mind of somebody like the doctor here, who understands that “the very best we could with what we had” is far short of “the best medical care we’re capable of.”
"Normally I’d rather not be operated on by a surgeon so strung out that"
Normally no one wants to be part of a "mass casualty event" but if one ends up in one, I would be glad if competent people do the best they can and not stop on arbitary regulations meant for normal times.
Absolutely, and I hope that I didn’t give the impression otherwise. In times of crisis I’m immensely grateful that most everyone will step up beyond their normal responsibilities.
I’m even more grateful that leaders like the author have spent their careers developing the experience, instinct, discernment, credibility, and fortitude to make the right adaptations under pressure.
I’d even agree with the original comment that a good leader trusts their people and gets out of their way. I just think that manifests differently in steady-state operations than during crisis.
The balance here is what I think is really impressive. They both built assembly lines AND rely on deep, adhoc experience & skill in applicable places. Efficiency in narrower, skill-specialized areas and an "artisanal" approach to initial triage.
Makes me think of the apocryphal story that the genius behind Sears Roebuck & co, the thing that enabled them to change from being "just another mail order company" to "The mail order company" was to invert the fulfillment process. this changed the cost of assembling an order from n to log(n)
I don't think it was Mr Sears or Mr Roebuck that came up with the idea, One of their warehouse managers I expect. I always sort of wonder why Sears did not become what amazon is, They already had the infrastructure for it. my best guess, unable to shift mediums fast enough?
While looking for sources to this story(I could not find any, sorry) I did find a claim that bezos was pivotal for a 1992 ruling that no sales tax was due for orders that originated outside the state. and this was critical for amazons early success. I worked for a mail-order company when that ruling was overturned. And it was a mess, I was not in accounting so I have no idea what it did financially, but all of a sudden all the software went from having to handle (in state collect tax, out of state no tax) to (every state, county, city and their dog charges a different tax and now you have to try to figure out what this is and collect it).
I thought that law only required you to collect state tax, if you were out of state, local taxes were still exempt unless you had an actual business in that state.
It was South Dakota v. Wayfair, Inc. I was a sysadmin so only tangentially involved, but the programmers were having a hard time because the situation was basically unknowable, That is, there was no primary source of information, I think they ended up buying a service that specialized in keeping track of what address charged what taxes.
The basic idea is that some localities charge a use tax, which is a sales tax applied to items bought outside their jurisdiction. The citizens if the jurisdiction are required to self report and pay this tax. The problem(or beneficent depending of what side of the transaction you were on) is that this is very nearly unenforceable, just too large an attack surface. The standard way to enforce tax payment is to move it a rung up the ladder. The store collects collects the sales tax before you get the item, your employer collects the income tax before you see the money etc. now there are exponentially fewer collection points. the collection points are hidden, out of the voter base and each point has far more to loose if they rebel keeping them in line.
I think it is that instead of sending a boy around the store to assemble the stationary order, the assembler stays still, only responsible for what is in reach and the order moves on conveyor between assemblers.
I don't think giving nurses unrestricted access to narcotics is giving them 'freedom to do their job' because a valid part of a nurse's job is controlling access to narcotics. It's not a medical need, but it's an operational and societal need. Same with whatever a CT tech would do away from the controls. I think this is more accurately a descoping of the job to improve throughput.
Depends how you define the job. A different interpretation is that the job of a medical professional is to improve the health of their patients as much as they can. Normally this is done by frequently having someone/something more specialized ensure perfect care for the task at hand, but on this night that might take too long, so the best way to improve the health of the patients was to abandon protocol and do the best they can as fast as they can.
These are core principles that are taught to Emergency Managers in the Swarm Leadership method [0].
Unity of mission – it was for these leaders, Save Lives!
Generosity of spirit and action – these leaders and people across the community were willing and eager to help one another.
Stay in your lanes, doing your job, and help others to succeed in theirs. How can I make you a success?
No ego – no blame. No one took credit for their success together. No one pointed fingers when problems arose.
A foundation of trusting relations – these leaders knew and had confidence in one another.
Giving people freedom to do their jobs is how we got the Crowdstrike incident. Those rules and restrictions at a hospital are in place precisely because there are enough people who cannot be trusted with the freedom to do their jobs to make them unavoidable.
Your premise is highly debatable; I don't know anyone who determined Crowdstrike was the result of "freedom to do your job", but regardless the hospital rules are for regular operation, not black swan events like this.
This could be a really cool use for AI. We have so many rules and systems in place to protect normal people from professionals that we have to blindly trust. If I had a reliable AI in my pocket I could be in charge of my own safety again.
I did this a month or so ago when a Dr. recommended a cortisone injection behind my ankle. I asked GPT about it, it said consensus recommends against it because it can weaken the Achilles. The doctor hated it but I am really glad I didn’t just blindly trust him.
I guess I like it because it is enabling instead of replacing humans.
> did this a month or so ago when a Dr. recommended a cortisone injection behind my ankle. I asked GPT about it, it said consensus recommends against it because it can weaken the Achilles. The doctor hated it but I am really glad I didn’t just blindly trust him.
Taking medical advice from ChatGPT over what your doctor says, what could go wrong?
Regarding your interaction with the doctor, I completely understand both sides, and I can guarantee he was well aware of the small risk of tendon weakening.
People usually want their physician to do "something", otherwise what's the point of going? This puts them in a no-win scenario. A single cortisone injection provides immediate relief, with a very small chance of side effects. If he doesn't offer it, people will complain about him being useless. If he does, some patients will think he's incompetent.
I'd find it exhausting having to second guess every interaction.
