This article is worth reading. The ideas are good, and the writing is clear. As in:
An error of omission occurs when an individual or organization fails to do something it should have done. For example, when Kodak failed to acquire Xerox when it could have, or when Xerox failed to develop the small computer produced by its employees. Of the two types of error, errors of omission are usually the more important. The deterioration and failure of organizations are almost always due to something they did not do.
...
Now for a key fact: Accounting systems in the Western world only take account of errors of commission, the less important of the two types of error. They take no account of errors of omission. Therefore, in an organization that frowns on mistakes and in which only errors of commission are identified, a manager only has to be concerned about doing something that should not have been done. Because errors of omission are not recorded, they often go unacknowledged. If acknowledged, accountability for them is seldom made explicit. In such a situation, a manager who wants to invoke as little disapproval as possible must try either to minimize errors of commission or transfer to others responsibility for those he or she makes.
And also this: Learning how to learn is probably the most important thing an organization or individual can learn.
In such a situation, a manager who wants to invoke as little disapproval as possible must try either to minimize errors of commission or transfer to others responsibility for those he or she makes.
The best way to do this is to do nothing, or as little as one can get away with. This is a major reason that organizations do not make radical changes.
>There is nothing wrong in making a mistake, but if you ever make the same mistake twice, you probably won’t be here the next year.
Some companies will accrete rules after every mistake until it is impossible to do anything.
>For example, when Kodak failed to acquire Xerox when it could have ...
I don't think it has ever made sense to buy Xerox, as an investment their record has been dismal.
>Kodak is currently in a precarious position, because it did not press the development of digital photography.
Except Kodak was a pioneer in digital photography. Their problem wasn't that they let the transition sail by. It was that they didn't recognise that they were a chemicals company rather than imaging.
Yeah I don’t agree with never making the same mistake twice, especially for lower level employees. Deming in The New Economics points out that all processes in an organization have natural variation, so mistakes only have meaning if they go beyond that variation in their impact.
So if your code is so fragile that it requires manual testing of every feature before deploying, it’s very likely that a junior employee will forget that testing step more than once, because the system has set them up to fail. Deming would suggest never blaming individuals, and instead measuring the process’s variation and working to reduce it, in this case by reducing code fragility or finding ways to automate or enforce the manual testing.
A couple years ago I did a deep dive into systems thinking. Two people whose ideas I highly respect, John Cutler and Will Larsen, reference ideas from systems thinking, which is what piqued my interest. In the end, what I found is that it wasn't that useful for the problems I face as an engineering manager, and I am skeptical about whether it's formally useful except in a very casual way.
The key work in systems thinking is drawing causal loop diagrams to identify potential feedback loops, some of which tend to stability and some of which tend towards instability. The way you draw these has almost infinite degrees of freedom, and so while they can sometimes be helpful in eliciting your own ideas, the outcome is ultimately heavily grounded in your preconceptions of what's important and about the relevant scope of the exercise. In working with it, I never had a sudden realization that some neglected factor was the key to everything and would provide previously unexpected levels of leverage. I never identified a feedback loop which provided outsized control over the process that I couldn't identify and attempt to resolve with traditional tools. So, as an individual analytical tool, I don't think it added much to deep thought, an outliner and a notepad.
The case studies and the literature about the practice emphasize its importance as a tool for communication and collaboration. It seems to me that many of the practitioners are mostly trying to use it as a rhetorical tool to try to win arguments about which they've already decided their bottom-line opinion. But I think in the context of a business, it fails at this in lightweight terms (i.e. without a major top down organizational push) because the idea is so foreign to others. First you would have to teach them what a causal loop diagram is, which is itself a quite nuanced topic, then you'd have to convince them that your particular construction and emphasis is the one that's most relevant for a decision. The "success stories" here make a lot of money for the consultants that are able to go do training for 5 layers of management like this author, but no one ever adopts it and makes important decisions which they credit to the incredible causal loop diagrams they drew.
Two additional issues. Systems thinkers sometimes emphasize the importance of quantitatively modeling the feedback loops. For almost all the things I care about, that's impossible or admits to the same explosion of degrees of freedom as the loop structure. If you decide that code quality is a concern, or that a deteriorating dev experience could be impacting velocity, you could try to find metrics that capture those, but finding metrics that capture those AND act as inputs or outputs to further nodes of the causal loop is pretty much impossible. Qualitative aspects of a system are of critical importance, and you ignore them at your peril.
Second, systems thinkers are not very good at thinking about probability and risk. The causal models allow you to think about what happens assuming you know about the inflows and outflows of systems and processes, but they can't be readily combined with an understanding of your own limited knowledge, or risks that are ever present in every decision. Thinking about risks quantitatively, even in ballpark terms, I found way way way more useful to my decision-making than all the time I spent thinking about feedback loops. Knowing whether your confidence that an improvement will work is 30% or 70% is directly useful, even if its only your informal probability, assuming that you are reasonably well calibrated.
Systems Thinking goes beyond just causal loop diagrams, which are a useful tool in soft-systems modelling. Systems Thinking is more of a foundational mindstate rather than a set of tools and I think this is something I disagree with the author on.
Systems Engineering uses Systems Thinking as the root of the apprach but will use any tool available for actually modelling/designing the system. A big one which is used a lot for hard-systems is SysMl (an offshoot of UML, I know) which is again somewhat inpenetrable for outsiders. I think thats ultimately a trap for all domain-specific tools where they need some way of being abstracted for the layman.
For an Engineering Manager, a combination of tools from the Operations Management disciple and Enterprise Systems Engineering is possibly something that might be a bit more useful to you.
Can’t you quantify the risks and add them to the inputs of those models then?
Same with code quality you mentioned, you’re effectively saying that the ‘feel’ or intuition about the code quality and on when to act to do something about that code quality, is more accurate than if you’ve tried to quantify the code quality and then quantify its effects on everything else and then try to use that as one of the inputs of the model.
I’m just asking all that, I’m myself relying on intuition a lot, but I feel like I always have a communication barrier when all I have is intuition; i.e. how do I explain my reasoning/decisions if all the risks and potentially positive outcomes are just weights in my head (that’s what I imagine intuition is)
The formalisms of systems thinking don't really work when you try to incorporate uncertainty. You have to frame things in terms of stocks and flows, but risk and uncertainty resolve in sudden leaps, not incrementally. If you assign 30% probability to "will have an incident" it doesn't smoothly climb from 30% to 100% over time, you roll the dice it jumps discontinuously to 100%. I'm not saying that attempting to quantify code quality and its impact is not useful, I'm saying the tools of systems thinking don't add anything to that exercise. Even if you gather data on the quality of your code, you still use expertise and intuition to judge those metrics, because they are always weak proxies.
To explain intuition, I do think it's successful to list the pros and cons and be explicit about which ones you think are low-medium-high likelihood and low-medium-high impact. Then people can disagree regarding your richer model.
Yup this stuff is not for managers. It's on par with expecting managers to work out maxwells equations by themselves. It's not what managers are hired to do. Better option is to hand over data to academics/experts, and do what you can until they return with better models.
Too often, it's human nature to bias a brand or idea with the reputation of someone associated.
McNamara was too much of a nerd in the inability to apply real-world critical thinking. He probably looked for a wife using a punchcard program in FORTRAN on an IBM mainframe. Hell, he looked more IBM than IBMers. Combined with the unpopularity of the war, all fruits of his efforts were poisoned.
If, say, Ross Perot had been SECDEF, perhaps the war would've ended sooner and the MIC wouldn't have been handed as much money, and possibly could've introduced McKinsey-like thinking.
> Mistakes will be forgiven if we learn from them.
I tell my kids this all of the time. For some reason, they haven't embraced this - I suspect it has to do with conditioning from peers, school, society at large.
> This was a typical organization, one in which the principal operating principle was “Cover your ass.”
Yep, sounds all too familiar.
I'm not going to quote every bit, this is a pretty insightful article.
Using jargon to mask a lack of understanding is also key. It’s hard to push back on a bad idea when you need a full day just to understand whatever buzzwords are being used to describe it.
I asked ChatGPT to describe a bad idea in technical jargon, storing all data in a single node in memory, and the result mirrors real descriptions of systems I’ve had to try to refute.
“We centralize all data in-memory on a singular node to alleviate the pitfalls of distributed storage systems, by employing a single node design pattern alongside cutting-edge memory caching algorithms to ameliorate performance, scalability, and operational efficiency. This transformative approach has the potential to streamline the data retrieval process while concurrently reducing hardware requirements, and augmenting compatibility with existing infrastructures.”
Something I've been saying is that the "asshole CEO" is necessary to get large orgs to actually do things. The author tells a relevant example, where a CEO is too timid to force change without the approval of his underlings (wtf?), who in turn are afraid to make changes with the CEO's approval. This results in a deadlock where nothing changes.
The kind of systems thinking outlined in the article is a very rare thing in the world, and outright frowned upon as "not proper" by those entrenched in the more conservative approaches to leadership.
Case in point: Elon Musk.
Compare SpaceX to NASA. The development cost of Starship, including about a dozen test articles, is the cost of a single SLS launch. People love to go on about how SpaceX is "making mistakes" and that their "rocket blew up", while neglecting to mention that the first SLS test launch also failed to achieve the desired orbit.
Elon Musk is one of the few corporate CEOs that get that mistakes are necessary stepping stones on the way to success. He encourages experimentation and rapid iteration, and will fire idiots that push back against improvement. Remember the engineer that argued with him about Twitter's slow performance, where it takes tens of seconds for it to show less than 1 KB of text? That guy had no incentive to make changes and was happy to let things be.
Meanwhile, I've been to over 150 organisations as a consultant, and not one of them has this mindset at any level of management. Everything has to be perfect up front. Everything needs to be planned 6 months in advance. Nobody is ever allowed to flip a harmless switch "just to test" an idea, no-no-no! That's not proper. The switch flipping has to be documented. It has to have a test plan, a rollback plan, a UAT plan, and seventeen people need to have a hundred meetings over months instead of simply trying the thing to see if it works.
This is the inherent nature of self-organised bureaucracies. Everyone optimises for their own personal local optimum, which is never the organisation's optimum.
Only the CEO can force behaviour changes to redirect efforts towards the global optimum, but that requires forcing people out of their local optimum, which is unpleasant for those people.
That's why you need "asshole CEOs" like Bill Gates, Steve Jobs, or Elon Musk. These people can never rise to top organically, via ass kissing and "making nice". They have to be there from the beginning, holding the reins of power in a way that cannot be taken from them by people that are made uncomfortable.
Your comment is the epitome of the faulty thinking mentioned in the article.
The essence of your post is: "Mistakes shouldn't be made, because there may be consequences.", which is the exact thing the article is warning against.
How many people died because of Tesla auto-pilot over the last decade, across hundreds of thousands of cars on the road? Half a dozen? A dozen? This must be stopped or at least regulated, that's your argument.
To put some perspective on things: About the same number of people were killed by gun violence in Texas alone just yesterday. According to half the voters in the United States, regulation can never be justified as a solution to this carnage. Never. Literally "any" regulation is "too much" regulation in their minds.
Systems thinking is about rising above these primitive self-interested, myopic decisions. It's using numbers and facts at a global scale instead of an individual scale.
If Tesla develops a working self-driving AI, they'll save tens of thousands of lives annually, if not hundreds of thousands. Killing a few people due to imperfect intermediate stages on the way there is worth it. Not to the dead people and their families, of course, but to the overall population.
It's uncomfortable locally but optimal globally. That's literally what systems thinking is about: forcing people out of their local optimums, against their will, to achieve a global optimum. That optimum might actually be preferred by everyone, but local gradient descent by individuals will never get over the hump to achieve it.
Just look at the people suing the FAA for allowing the Starship launch. There are people that would rather that we remain stuck on Earth forever than take the slightest risk, like a few chunks of concrete being thrown around. It's absurd, but its the exact behaviour I see in large organisations at every level.
You seem to have taken the article's point of taking FINANCIAL risks to a company's bottom line, and perverted it to justify killing a few of your customers for the sake of a beta test. Why not go for human medical testing without consent, while we're at it?
Taking risks MUST have limits, especially when it comes to human life.
Have you not seen Fight Club, or watched the news about the 777 MAX crashes caused by MCAS?
Big corporations take risks with their customers lives all the time. There's an entire area study dedicated to calculating how many dollars people are "worth" when making such risk calculations.
Virtually nothing is risk free. Anything that involves anything you ingest, anything that can catch fire, anything that involves driving, and anything that involves mains voltage can and does kill "customers". Virtually nothing of import done at an industrial scale has a perfect safety record.
Solar panels kill people: again, workmen fall off roofs while installing them and break their necks.
Should we stop all progress on green energy because it is totally unacceptable for any corporation to ever risk anyone's life, ever? Is that a net benefit? Should endless studies be done, for centuries if need be, to make sure no windmill ever kills a worker, ever again?
hmm... So, ethics panels are not needed? What is a few lives in ways of progress? FAA and spaceX circumvented existing process - there was a larger article that this is going to be an ecological disaster before the launch that was posted here [0].
You are willing to sacrifice what is in your hand now for an nebulous future promise and that seems to be equally faulty bias on the other side. Almost all fraud happens with a vague promise of future gains, if only you can sacrifice a little now.
What differentiates false promises from true hope? How much would be risk on a too unbelievable outcome? These kind of "looking from hindsight" articles are pretty useless. Every idea that has to be funded also needs to have a realistic chance of success at that point in time.
That blog your linked is packed with emotional speech and hyperbole.
"Noise bomb" is one of the literal headings.
It's a rocket launch pad, what did they expect, a gentle susurration whispering through the reeds of the uninhabited wetlands!?
This anonymous "ESG Hound" character posts virtually nothing but rabid anti-SpaceX rhetoric. I bet he doesn't even live in Texas. (For reference, the locals, when interviewed, said they welcome the huge uptick in business SpaceX brought.)
This particular unhinged rant is about the sound levels, saying that "110 decibels" can cause structural damage in nearby towns, etc...
First of all: no, it can't. That's absurd.
I live 2.5 miles from a sports stadium where they semi-regularly host rock concerts that go up to 120 dB. At that distance, that's louder than when my neighbour turns his stereo up to 11. This insanely loud sound goes on for hours on end, but it is perfectly legal, and that stadium has apartment blocks a mere few hundred yards away from it!
Second, whatever "ecological impact" SpaceX had on the area, it pales in comparison to everyday, ordinary industrial activities. Building demolitions, mine blasting, road works, F1 races, etc...
Elon rightly pointed out in an interview afterwards that the area is prone to tornadoes and hurricanes, which do vastly more damage. No human activity can even hope to match what nature can do, short of nuclear weapons testing.
What actually happened is that a few chunks of concrete were strewn about, an activity accompanied by a noise so awesome that people flew there to get the chance to experience it!
> What differentiates false promises from true hope?
Who said everything had to be certain? So what if SpaceX fails and we have nothing to show for it except some deafened seagulls and concrete littering a beach?
Should we never strive for the stars just because to get there we might make a mess?
Who are you to nail humanity's feet to the ground?
No, the risk landscape is not binary. The choice is not, take no risk or take any risk. There are literally factors of millions if not billions between the low risk and high risk activities. The people arguing that the status quo level of risk tolerance are too low almost always mischaracterize how wide the gap is or do not know how safe the “risky” activities they point to as evidence that risky activitys should and can be tolerated. Yes, you should weigh the pros and cons, but that requires actually quantifying them.
For instance, you pointed to the 737 MAX in a different comment as evidence that corporations take risks with their customers all the time. First of all, they were roundly criticized for their laxness, so that is actually evidence that too many risks were taken and we should stricter. Second, the fatality rate of the 737 MAX right after the second crash (while it was defective) was on the order of 1 fatality per 100,000,000 miles which is still safer than driving. Third, the reason it is viewed so negatively is because that level of safety is around 1,000x worse than the average in commercial airliners.
Arguing that we accept corporations taking risks because we accept the 737 MAX (we do not), so we should increase our risk tolerance to the level of the 737 MAX (everything except other airplanes is already more dangerous, so this would actually be decreasing risk tolerance) and generally accept greater levels of risk than the status quo (which has been rejected in the 737 MAX case as we find the increase in risk relative to the status quo unacceptable). The entire argument is literally completely wrong in every conceivable facet.
To then dig into your other specific argument, the Tesla argument is that they are taking greater risks to develop faster. A pro and a con. However, by all objective measures, Tesla will not develop a self-driving system first. Their systems are thousands of times worse than their competitors such as Waymo and Cruise. Their approach is so far behind their competitors that there is little reason to believe that anything it is doing will be incorporated into working solutions. So, every single person their human experimentation program kills, of which there are at least 31 documented deaths (no thanks to Tesla who deliberately obscures reporting), is being killed for no reason. From a systems thinking perspective, there is exactly zero benefit and immense costs to supporting Tesla’s regressive development process relative to others who are both more safe and more effective.
Nobody brought up Tesla, except obliquely by way of Musk. And arguably Tesla is another good example (electric cars, not self-driving). Do you have a response to the other names listed (Bill Gates, Steve Jobs)?
You can invoke survivorship bias on any entrepreneurial anecdote, so it doesn't contribute much to the discussion.
Also, the "asshole" was in quotes, and that's significant. No-one's discussing their social graces.
I know they are an utter failure around here in every aspect, but by the numbers Google has been running very successfully for a long time with employees having a lot of freedom and a "making nice" culture. Their biggest failure (Google+) was when one of the type you admire got too much power.
You are conflating boldness with assholery. You can be one without the other. But the problem is that assholery is what is most visible on the outside, so people tend to attribute any outcome to it.
Tim Cook and Satya Nadella for instance are good examples of bold CEOs who are not assholes. They have risen through the ranks organically over decades, and are widely regarded for making bold moves as CEOs and being successful at it. Satya Nadella in particular is credited for bringing about a cultural change within Microsoft and creating an environment where risk taking and learning are emphasized.
An error of omission occurs when an individual or organization fails to do something it should have done. For example, when Kodak failed to acquire Xerox when it could have, or when Xerox failed to develop the small computer produced by its employees. Of the two types of error, errors of omission are usually the more important. The deterioration and failure of organizations are almost always due to something they did not do.
...
Now for a key fact: Accounting systems in the Western world only take account of errors of commission, the less important of the two types of error. They take no account of errors of omission. Therefore, in an organization that frowns on mistakes and in which only errors of commission are identified, a manager only has to be concerned about doing something that should not have been done. Because errors of omission are not recorded, they often go unacknowledged. If acknowledged, accountability for them is seldom made explicit. In such a situation, a manager who wants to invoke as little disapproval as possible must try either to minimize errors of commission or transfer to others responsibility for those he or she makes.
And also this: Learning how to learn is probably the most important thing an organization or individual can learn.