Almost all software outsourcing I have seen end up costing more in reputation damage, drop in quality, bug pilling up, incompetent programmers, etc. World would be a better place if MBA managers were not running it.
Anyway this is why professional engineering were created, to protect the public against those exact bad practices. If we were serious about fixing the issue, we have the solution.
A key factor in such failure is misaligned incentives. A team in a company is more likely interested in solving problems for the longevity of the company, so they think long term and they iterate. They also learn about the domain over the years. On the other hand, outsourcing companies just want to get a project done in time and under budget, and their hire the least qualified people to just meet the negotiated bar. To that end, there's really no incentive for an outsourcing company to hone any kinds of craftsmanship (there are exceptions in small and specialized consulting firms, of course). Plus, there's always information loss and distortion when a company communicates their needs to an outsourcing vendor. Consequently, all hell will break lose when a project is complicated enough.
It is not necessarily outsourcing fault but rather “people is a resource” mentality. This mentality leads to outsourcing to cheap locations because they are cheap. But I’ve seen in other context as well.
The problem with any professionalism setup is that software engineering is a very broad field whose use cases range from "it doesn't matter, it's mostly for fun" and "this software runs the international space station".
On the one hand, there should probably be some sort of certification or liability if you're writing code for NASA or Boeing or Airbus or maybe the New York Stock Exchange or what not, where a mistake could cost billions of dollars and put lives at risk.
At the same time though, there's also stuff like the video games industry, people developing programs for fun, blogs and fan sites and things that are free and mostly static, video game mods and hacks and about 99% of small business and organisation websites where frankly it doesn't matter that much how well they're coded or how stable they are or what not.
If my bank screws up and gives away all my money, that's a problem. If my copy of Tears of the Kingdom breaks, that's not a big deal (and might even be a benefit).
If there's any sort of professionalism or certification or requirement, it should probably be limited to industries and fields where the consequences of screwing up are life changing for a high number of people.
MBAs are the worst thing that has ever happened to humanity. Their profession is the metaphysical and social equivalent of cancer: They exist, they consume, they grow. Nowhere at any point does it seem to matter that everything dies if we let it go on untreated.
I think that describing the MCAS disaster using predominantly SW Engineering language is misleading.
The substantial change that might have averted the catastrophe is having qualified engineering oversight integrated into the MCAS project management structure.
MCAS is a flight control application; its defining discipline is control theory. Thus the hypothetical engineer who could have averted the catastrophe would have had to be a controls, rather than a SW person.
I have read much about MCAS, but no detailed narrative ever mentioned a SW bug, which implies that MCAS SW has apparently functioned according to given specifications.
Thus, while $9/hr SW engineers is a choice correlated with an inadequate safety culture, I fail to see the casual link between that and the tragic outcomes; I cannot condone asserting such causation.
And the fact that they wanted to spend nothing on the software and then rush to production (this was exposed in this article) is some cluster-b antisocial personality disorder shit...The bean counters had the stock churning at all time highs and they were going to do things their own way (trading at 450 per share shortly before these tragedies).
So it's HIGHLY unlikely that Boeings software for the 737 Max would have been operating at spec given the use-case (overcoming hardware problems with software in a way that's never been done before). MCAS was not designed to take into account that external sesors could be out of wack (which happens all the time). MCAS took liberties and had opinions that ran counter to the norms of aviation. And this is the hallmark of poor software engineers that have no domain expertise and no ability to push back against anti-social personalities masquerading as managers.
"
When the two computers disagree, the solution for the humans in the cockpit is to look across the control panel to see what the other instruments are saying and then sort it out. In the Boeing system, the flight management computer does not “look across” at the other instruments. It believes only the instruments on its side. It doesn’t go old-school. It’s modern. It’s software.
This means that if a particular angle-of-attack sensor goes haywire—which happens all the time in a machine that alternates from one extreme environment to another, vibrating and shaking all the way—the flight management computer just believes it. "
If you want to have the title of SR. Engineer and lives are in the palm of your hand you better be prepared to be the bad guy and take on management when they're being driven by motivations that run counter towards the quality and the usecases for your code. VW had a similar culture and those "senior software engineers" went to prison, not the managers (remember that).
This. On top of the MCAS modification being necessary in the first place.
Higher torque generated because of more off-center thrusters surely raised eyebrows in the stall risk mitigation team.
In the end, that's 3 teams where management made the topic being fixed by another team. That's systemic. And with a single confounder: cost measures being prioritized over safety.
With the current QA problems, it cannot be made more clear that that single culture element is the root cause at Boeing. And ethics dictate that no more experiments be made, and the top 200 of management be forcefully removed.
The other problem from what I can see was a lot of frog-boiling. MCAS was actually originally designed for a slightly different aerodynamic situation, but in flight testing, they discovered a different and more serious instability.
The engineers determined that they could use the MCAS to fix this problem too, but the amount of control input had to be about 4x as much for this situation (2.5 degrees vs. 0.6 degrees of stabilizer movement). They also made it so MCAS could activate multiple times. The original safety analysis was based on the 0.6 degrees for one shot, which wouldn't have put the plane in a situation where the pilots couldn't overpower it, like what happened in the two crashes, so the system was put in a lower safety category that didn't require the same redundancy. Dominic Gates wrote about this in [1]
Given that the plane was already built and in flight test, the production lines were ready to start, and there were massive contract penalties for either late delivery, or additional pilot training, the pressure to hack something in and ship must have been enormous.
The only place to make a change and still hit the deadline was in the software, and unfortunately even the software was limited by Boeing's redundancy strategy of having two completely independent sets of flight computers and sensors. Having one computer look at sensors from both the left and right sides compromises the concept of having them be completely independent, and so it wasn't done, even though ultimately that was what was used as the fix.
Apparently 1/3 of all software vulnerabilities represent design weaknesses which were introduced in the requirements phase. The MCAS flaw seems to belong to this category which you describe.
Separating controls from SW is a strange thing to do. The issue here is a culture of shortcuts and awful money-driven decisions, not any particular discipline, though delayed software fixes did cause more accidents, and it was poorly designed overall.
Ultimately, the MCAS relied on a single sensor, one which was known to fail, and only displayed the redundant sensor data - get this - if they bought the additional option to show when the sensor failed.
> However, whereas MCAS was activated automatically, without pilot action, the cockpit crew would have to notice and act on an AOA DISAGREE alert. Further, the AOA indicator and disagree alert were not standard equipment on the 737 MAX, although the AOA indicator had been on earlier models. Boeing offered them as “add ons” at additional cost
Without that "add on" you'd never know the MCAS was acting on faulty data. And all the while management worked overtime to mislead regulators on the potential impact of MCAS to avoid additional scrutiny and training requirements. So nobody knew the safety of their aircraft depended on that non-standard equipment package.
Yeah - total business and design mismanagement driven by greed.
> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004
I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
>> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004
> I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
That is the MBA cancer. Focusing on shareholder value instead of making a great product.
I've come to the conclusion that just like we require licenses and certain degrees to work on safety critical fields we should extend that to things you can't have in order to work for companies that do manufacturing, starting with MBA's are legally ineligible for hire at companies like Boeing.
Yeah, I suppose like the biological cancer, there's not a clear cause effect relationship between a reshaping culture, and then MBAs or "Ship-it" mentality taking over engineering oversight and scheduling. It's a runaway effect.
I tend to place more blame on a deliberate reshaping by the top individual, rather than some accidental metastasizing of the problem. I'm straining the analogy.
I've always seen the cancer as MBAs only hire MBAs, and that's how it grows. Its very difficult to get to a senior level in most organizations without an MBA.
Maybe the CEO in such cases often is an MBA and you are a bit talking about the same thing, just from different perspectives? (Looking at the one person, vs the people, at the top?)
> I have compiled the following for the 2021 Fortune 500 US companies (the last global one I've seen is from FT in 2015 https://ig.ft.com/sites/mba-to-ceo/):
> 43% of CEOs have an MBA
Anyway,what happens if the CEO is an engineer, and everyone reporting to him/her is an MBA :-)
Not arguing anything in particular, just want to point out it's not completely unusual to be both. Many of the top "Chief" positions I've worked under were former PhD engineers and researchers who went and got an MBA to move up. The legendary director of JPL, Charles Elachi, for example. https://en.wikipedia.org/wiki/Charles_Elachi
And many of my former bosses.
Maybe that's a symptom of the same disease ("without mba you cannot rule"), but I think it doesn't necessarily mean an MBA holder is a bad candidate for leadership.
I remember reading that development of the MCAS system was not done by offshore companies, which was confirmed by Boeing. So saying that $9/hour employees caused the 737 MAX issues is incorrect.
>Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.
>In a statement, HCL said it “has a strong and long-standing business relationship with The Boeing Company, and we take pride in the work we do for all our customers. However, HCL does not comment on specific work we do for our customers. HCL is not associated with any ongoing issues with 737 Max.”
>Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.
It's a super fun read. And you're absolutely right: Boeing caused the problem itself. Shifting blame to poor software "accidentally" produced by 9/hr engineers is not legitimate.
There was a bug. But it was only a problem because all the other redundancies and safeguards were shortcut and regulators and pilots were mislead about the severity of MCAS system's "corrections" and not given proper data to know when to override it.
I seriously doubt this is the mistake of $9/hr workers. Those workers are unlikely to be familiar with the whole picture and would probably just have requirements delivered to them. This is not a case of badly written code, the code didn't crash. It just didn't fit the requirements, likely because those requirements weren't known, because someone I'd wager working directly at Boeing getting a industry-typical salary didn't deliver them.
Part of good engineering is having a solid understanding of your business domain and being able to spot things that just don't smell right.
An experienced professional software engineer on an industry standard salary, having been provided with nonsensical or missing requirements, should know when to impute and when to raise an exception. I'm not at all confident that a $9/hr engineer will do that.
Well, we don't know if they did. We do know that Boeing employees (including one of the whistleblowers) have attempted to raise such issues internally for a long time, but mostly went ignored, moved to a department where they can't complain, or fired.
So purely on the power differential, a $9 engineer wouldn't make a stink. Says nothing about their work ethic or ability to see disaster (at least upon integrating code, probably not literal disaster) coming.
And also, how would they have known they're just using one AOA input, and a component above them isn't already delivering a value that's known good - these are not the kind of engineers that'd know that.
I'm not at all confident that a $9/hr engineer can do that.
You don't hire somebody for a price like that because you intend to listen to their feedback and treat them like a valued member of the team. You hire somebody for a price like that because your LLM hasn't quite reached the level you want yet...or frankly costs more to run.
Exactly. Seems just a bait article from someone who doesn't know shit about what happened (similar to Dave Farley video blaming the Cyberpunk launch fiasco on the lack of CI/CD).
You outsource your software development to some country with quite low average monthly salary. Say some country in eastern Europe, where average salary is $500 / month.
Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month. Given 150 work hours in a month, that comes out to $10/hour gross pay.
You get good engineers, pay less, and they earn more than they'd get from pretty much any domestic employer.
If the average software dev in the US makes, dunno, $70k / year, then that would be the equivalent of getting hired on a $210k / salary.
Of course, it's not all smooth sailing - but I think the important part here is to keep in mind that pay is relative. What could be a pitiful salary in rich western countries, could be a very good salary other places.
You are underestimating how difficult it is to communicate with people from different cultures.
I did an electricity project with developers from the Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.
I also worked with a Ukrainian graphic designer who had no understanding of copyright law.
> You are underestimating how difficult it is to communicate with people from different cultures.
US devs are highly overestimating the value of being American. They don't understand that a Ukrainian dev working at 80% performance because of cross-cultural issues for 50% salary is still a good deal. And when the company becomes international then the whole argument becomes reversed because the US devs become a minority and it's them who fails to adjust to others. Sure, it's difficult to hire a good dev from abroad, but when this gets done right, it saves a ton of money. For example, an American company can buy a smaller, healthy foreign company, and with the company, all the knowledge of operating in the local job market.
There are tons of posts on Reddit "I'm American and I graduated in Computer Science and I was promised $200k jobs but I can't get hired at all" from people who fail to understand basic economics of big companies.
Not necessarily, because the bottleneck ends up becoming the Americans once again. Because the business is here, they have to be the ones to identify and fix mistakes, as well as do culture translation. So you end up saving employees of one category but then having more of another category.
One of my professors used to work at RadioShack and they had large teams in India. The software quality was good, but the interoperability was really, really bad. Not surprising since communication was awful due to time zone differences.
She, and her team in America, spent a huge amount of time normalizing their output to the American output. And you have to go in that direction because the company is in America.
> Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.
What are you talking about? Private apartments and houses in Ukraine had domestic electricity meters installed since soviet times.
> Say some country in eastern Europe, where average salary is $500 / month.
Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month.
This is 10-15 years out of date. And even then it was hardly ever as straightforward as that. And only ever applied to junior and maybe mid level developers or those who couldn’t effectively communicate in English. High skilled one were significantly more mobile and were basically competing with a much more global pool of developers. They could relatively easily move (if not to the US then at least Western Europe so your potential savings were usually limited to the difference in CoL + some premium).
This led to pretty high inequality based on skill/experience e.g. to top CS graduates could expect their income to increase by 3x if not more over the next 5 years after graduation.
Average wages for workers in Eastern Europe are way higher than $500/mo, never mind software engineers. Poland is $1,400, Hungary $1,200, Bulgaria $1,100, Romania $1,600, Lithuania $2,200, Estonia $2,000, Latvia $1,700. If you assume that software engineers will earn at least double the average wages of all workers, you're looking at $2,200-4,400 as a base. Paying 3x that to get the top people would mean $6,600-13,200/mo. Ultimately, you'd be paying $44-88/hour to execute your strategy.
Countries like Bulgaria and Hungary are both in the EU allowing workers to travel to Germany, France, etc. to get jobs. They aren't trapped behind immigration laws that don't allow them to move to places with higher salaries.
Even if you look at India, $10/hour wouldn't get you the best engineers. Mediocre engineers are earning that after a few years experience. Maybe you'll say that you'll pay triple that and that $30/hour is still cheap. Sure, $54,000/year is a cheap software engineer, but I think the best engineers have better options than being seen as a cost savings for some American company.
Frankly, it isn't necessarily about the money itself. It's about the attitude. If a company sees software as a cost center to be optimized/cut, their software will be crappy. If you're just focused on cutting costs, it's not just about how much you're paying engineers per hour. It's also about whether you let your engineers work on something that makes the product better, but might not be strictly necessary. For example, maybe there's a process the customer has to go through and many do it wrong because it's confusing. Will you let them spend the time to re-write it? Or will you say that it works so you don't want to spend money on that and customers will put up with it?
In theory, you could have more engineers and allow them to do more if they were cheaper. I've found that (in practice) when companies are looking to optimize salaries like that, they're also optimizing the time spent on things as well. They don't see their software as a strategic advantage to be invested in. They just see a cost center to be cut and that won't just be salaries, but also hours worked.
It also creates a crappy atmosphere where workers often don't care about getting things done beyond what they need to keep their job and not get yelled at. You're looking to work the system for yourself and minimize costs and they'll get a similar attitude where they try to work the system and do as little work for you as possible.
I'm not saying that you couldn't create a great company in Eastern Europe or India - they exist. But I think if your attitude is one of outsourcing a cost center, you won't get what you're looking for. Pay is relative, but it's higher than you think and you don't want to fall into the trap of just thinking that software is a cost center.
Plus, you don't even have to go that far to get cheaper engineers. You can pay half of what you pay Americans in France or Germany.
In some countries, they literally hire people off the street for coding who aren't even programmers, to sell them to suckers in rich countries overseas who will buy their services.
Any decent engineer worth their salt gets a visa to work in the said rich countries where they can earn 10x more. They don't stay in the local lower paying roles as they can't compete on salary.
I forgot which book mentioned that big government contractors like Boeing had to set up their factories and give business to vendors in many, if not all, the states no matter how inefficient or expensive that is because senators wouldn't approve key contracts unless the contractors brought jobs to the senators' states.
I'm guessing the cost of the outsourcing is also part of this compromise to the senators' demands.
A trajectory question: our government also hands out billions to NGOs and contractors to "solve" social problems, yet making such problems worse. For instance, multiple articles reported that how so many organizations charged ridiculous amount of money to SF government to help the homeless, yet the city's situation only got worse over the years. Wouldn't this be a worrisome sign of a society. An empire didn't fall because of external enemies but because the empire had so many entangled interests and the society simply declined and then collapsed.
The largest company in it's industry that uses many mission critical applications systems every second of the day, is in the process of offshoring all its software development to you know where.
When a company brings in an Indian CIO, it's only a matter of time.
The author himself notes paying at least a $100K global salary as of 2019, which implies he does engage in outsourcing. As such, the article is self-contradicting. Maybe the author should go back to writing about Javascript since javscript-scene is in the URL.
You can get bad engineers anywhere, including in the US. It's up to management to enforce a sane engineering process, although being the parasites that they are, they never will.
But also, lack of oversight, reviews, inadequate testing, etc.
I may not be $9 / hour but I still wouldn't trust myself to write aviation software. Nobody should have that kind of hubris, yet there's developers like that in all income tiers.
Here I would question the whole process. Do these developers have even any chance to stop this process or raise up issues? Is anyone listening to them? Or will they end up being removed if they raise up something?
But if you present your badly written requirements to a person who has the culture to ask questions, you at least have that safeguard. After all, it's a collaboration.
And when you work with an offshore team, quite often as contractors, you have the issue that you don't get top quality.
The in-between company will do its very best to hire low to maximize profit.
The good quality developers work either on local grown software, or leave the country for better opportunity.
And you are now working with a group of developers whose livelihood depends on saying 'yes' or not saying 'no.
And because of this you will not know whether there is an issue in the requirements until x time later and the plane goes down.
I really do not think the programmers are the responsible in this case. I'm not sure, but for what I understood, it was not a failure in the SW, but a failure to explain the pilots the changes in the behaviour of the plane.
On whole failure of the goal. Change a plane in big way, but try to get it act like it did not change. To avoid explaining and training pilots. So instead of training pilots to expect plane to act certain way in certain scenarios, it was instead fixed in software and hardware. But well that combination was done poorly and it should have still been explained.
Don't your investors oversee cutting of corners like recycling old designs to the breaking point? Or hiding critical new workarounds in the shallowest possible training?
"the sole fiduciary duty is to deliver profits" is a reductive meme at this point. The problem is putting short-term over long-term, and that's a choice Boeing executives made. Of course the system incentivizes this, especially if you get away with it. But they didn't.
Most shareholders hardly have a say on how major public companies are run. Thr Only signal they have is by buying/selling stock or not even that if you only own your shares indirectly through an ETF.
The more often capitalism strives to cost reduction and profit-maximizing at the cost of quality -- and sometimes lives --, the more often I think that companies should rely on non-voting shares far more often, with a promise to not blindly follow profits over all else.
I currently have very limited options if I want to invest my money into an (engineering) quality-first company.
> I think that companies should rely on non-voting shares far more often, with a promise to not blindly follow profits over all else.
sure, but how is that relevant? it's not like the previous CEO of the company was sacked by shareholders because they demanded a 0.05% increase in net revenue.
More slightly-ethical, hopefully sustainable capitalism. I liked my products pre-enshittification, when they lasted 20 years and were built like tanks.
Hark on them all you want, but I‘ve been using my 2016 MacBook for 8 years without problems.
> I seriously doubt this is the mistake of $9/hr workers. Those workers are unlikely to be familiar with the whole picture and would probably just have requirements delivered to them. This is not a case of badly written code, the code didn't crash. It just didn't fit the requirements, likely because those requirements weren't known, because someone I'd wager working directly at Boeing getting a industry-typical salary didn't deliver them.
Why do you think $9 an hour is shameful? Doesn’t that entirely depend on the cost of living? $9 could be 3x the average wage in some parts of the world.
I mean, there are a lot of things it could depend on. I'd like to say that it should depend upon the value that the programmers are creating--that more productive programmers should make more.
Unfortunately, if the article in the OP is true, these programmers were negatively productive :-( so it can't be as simple as just how much value a programmer produces. It's more complex than that.
Honestly, I've never gotten any good answer from anybody about what wages "should" be, for any job, programming or not. In reality, what happens most of the time is wages are simply determined by supply and demand, like anything else. There are plenty of people working for $9 an hour in very expensive cities in America--so whatever it depends on, it most certainly does not depend on standard of living.
If you take the article at face value--if its claims are true that $9-an-hour programmers cost Boeing billions of dollars--it is indeed a shame that, in this particular instance, Boeing was not willing to pay more for better programmers.
Are those claims true? Who knows. I'm sure that whoever hired those $9-an-hour programmers thought that if they could manage those programmers soooo well that they could get them to program as well as $100 an hour programmers.
YMMV, but in my experience, you can't manage programmers to be better programmers. No matter how much check-in approval bureaucracy, or how much QA, you will not get better programs. If you want better programs, you need to hire better programmers.
Well, for better or for worse, it is true that 9 dollar/hour or 1440 dollars/month would be a decent salary in India. Twice the current median.
I saw your other comments, so I would add that while it is a decent salary in India. That doesn't mean you can expect the same quality as you would expect from someone earning twice the median in US. Because standard of living never really scales like you want it to.
200 dollars in India still cannot buy you a cheap computer, while 1000 dollars (the equivalent in US) can definitely buy you a solid computer.
So, yeah, it is shameful for this kind of work, because it is skilled work, and skilled workers wages (because they can essentially move all around the globe) are not tied to cost of living.
Are you saying the CEO really wants to pay for high quality local software engineers but due to external factors behind their control they really can’t.
It's a selection problem. Public companies must make number go up no matter what, so they hire CEOs that do so. The CEOs that might focus more on sustainability eventually get filtered out of the system, much like an engineer that complains a bit too much about lacking security protocols.
That's why CEOs that have a history of destroying companies and making bad decisions keep getting hired. Because what they're good at is extracting wealth in the short term which is the only thing that matters.
I wish people wouldn't spread this nonsense. Public companies can care about whatever they want until their shareholders sack the CEO, which is an extremely high bar. It seems extremely improbable to me that there would be a shareholder revolt over "hi, we'll be increasing COG by a tiny amount to improve quality of the METAL BIRDS PEOPLE PAY TO BE PUT IN".
your blithe blackpilled acceptance of sociopathy is wrong and unhelpful.
Alright guy, I'll bite. Why do _you_ think that Boeing off-loaded manufacturing burden to 3rd party companies? Their executives have effectively gutted the company over the last 20 years. Now we're left with a bailed-out husk the government literally _cannot_ afford to let go out of business.
The shareholders choose the CEO to improve returns, not to improve safety, until safety safety impact returns (which it did for Boeing). Which is fair, we live in a capitalist world, capital is thus by definition deciding power and wanting to have more of it is a decent want.
Anyway this is why professional engineering were created, to protect the public against those exact bad practices. If we were serious about fixing the issue, we have the solution.