Boeing's 737 MAX Software Outsourced to $9-an-Hour Engineers (2019)
industryweek.comIt is absolutely a Boeing failure and not due to subcontractors/outsourcing as the clickbait title seems to imply.
If you parcel out parts of your job in something as critical as Airplane Construction, it is your responsibility to validate all specifications/testing/verification/etc. If you treat "Airplane Engineering" as mere "Embedded Engineering" you need to be put out of business.
The article explicitly says MCAS and "cockpit warning light not working" problems (which were the cause of the two crashes) were NOT outsourced.
Key points;
“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.
Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.
“Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s.
The last point is the most damning; Airplane Engineering cannot be commodity engineering.
No disagreements on your main points, but I would dispute calling that headline clickbait. We published this story in 2019 when software issues were the central issues at Boeing, so establishing how the company was coding was very newsworthy. I typically associate clickbait with bait-and-switch tactics, tricking readers into reading content unrelated to the headline. This article provides material directly related to its title. Robert Schoenberger, editor-in-chief, IndustryWeek
Glad to see that news websites (in this era of fake news) are willing to stand behind their stories; Much appreciated.
While i understand your point, the headline gives the impression that Boeing's problems stem from outsourcing which then emboldens the racists to come out of the woodwork (as can be seen from some of the downvoted comments here) and start manipulating the conversations in a different and unproductive direction. Some of the folks commenting didn't even read the article before dunking on the outsourced country/engineers. It is highly frustrating when this sort of thing happens and hence my warning in the first line of my response to ignore the title and read the article.
It would be great if you folks (maybe in collaboration with others sources/folks) could collate all the available info. on the disaster that is now "Boeing Engineering" (HW/SW) and publish a point-by-point synopsis of everything known until now. This would be a great learning lesson to the entire Industry on what not to do in pursuit of mere profits.
Your frustration is understandable, particularly in this situation where the critical failures weren’t the fault of outsourced engineers.
But we really need to pump the brakes on the whole “everything must be in the passive voice and overly euphemized to the point of nigh-incomprehensibility”. If anything, it contributes to the click-bait problem.
I see: “Boeing outsourced to $9/hr SWE” and immediately start thinking about a dangerous corporate culture that prioritizes profits over lives. I’d argue most people probably interpret the headline in a similar fashion.
Just because some already-inclined reactionaries wrongly jump to “India lol” doesn’t mean the publication has some moral responsibility to contort the headline into something less effective that will get even more dramatically outcompeted by click bait headlines that actually bait racists (“Deadly Boeing crash was running Indian software”).
In this case the publisher has a responsibility to not throw shade at outsourcing since it was already known that it was not responsible for critical software implementation. Whether by accident or design a lot of folks will draw the inference that i pointed out. You can also look at the previous articles submitted on HN on this topic to see how everything gets manipulated.
Outsourcing is a perfectly logical approach to saving costs. But the company doing it has to be careful on how it does it, have necessary modeling/tests/verification/QC/QA in place and be very very careful in outsourcing highly critical/specialized/domain specific subsystems (if at all doing it).
From all that has been published on this topic so far, there seems to be a systemic rot in the entirety of Boeing Engineering and Management which needs to be investigated thoroughly as a whole rather than piecemeal i.e. focus on the system itself rather than the simple cogs in the system.
> recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature.
Shame. Industrial companies, if you decide as pincher, you will be doomed in your little world... as it always has been :)
Most big "industrial-like" product companies are pinchers when it comes to SW. They offshore a lot hence why they pay SW devs so little. And it mostly works for them.
It works for them, because like Boeing said, they have old battles tested processes which usually means big mistakes can easily get caught ahead of time. And if big mistakes don't get caught they usually can't kill anyone and generate a huge public scandal.
I don't think it works for them, I think they are just entrenched with good moats.
What I think actually happens is they get to prove all sorts of ridiculous things are good management and that all sorts of managers need to be coming up with complex ways to manage what would be obvious in a situation where reality was still visible underneath.
In the end they pay a lot more than if they had competent individual contributors and less darwinian fights to the death in their org charts. But having a better organization is not really compatible with the incentive system involved.
I can easily say some of them calls themselves as "futuristic" pays too many unrequired attention and money much more than they pay technological infrastructure... what a oxymoron
The article seems to conflate engineering salaries in a low cost of living country with the quality of the software.
As I understand it, the software performed exactly as designed, forcing the nose of the plane downwards to counter the upwards torque produced by the engines being offset relative to the centre of gravity of the plane.
Mentour Pilot had a great episode on the two tragic accidents. It appears that Boeing did not expose the way the software was designed to behave, and the system silently turned itself back on even after the pilots disengaged it.
My experience is that contract work like this ends up being done by ordertakers; the contractors execute the design, no matter how stupid.
Whereas, the best performing companies hire engineers who are empowered to understand the goals, understand the problem, and push back when appropriate.
The results are much better, because the people working on each system, understand it deeply and care.
A low-cost contractor isn’t allowed to care. They aren’t even allowed to talk to somebody who matters.
This is a management disgrace.
Often this is the entire point. The in-house staff are seen as obstructionists, whereas outsourcers provide the equivalent of meat robots.
Many managers far prefer the latter over the former.
They prefer the latter because the mere existence of the knowledgable, informed, involved engineer questions the need for the manager at all.
I think that this a fair take, especially about the possible power structure.
Weird take.
The theme throughout has been cost cutting and poor oversight for the sole benefit of shareholder profits.
A proper in house team would probably say, hey what happens when the sole sensor feeding our software is obstructed by a birthday balloon? Maybe we need an exponentially backoff instead of every 30 seconds. Maybe nose down actions shouldn't be automatically executed at low altitude.
What we see playing out is the result of delegation and "not my problem", outsourcing safety critical systems to the lowest bidder...
I understand that it may be more likely for this to happen with an in-house team, but my experience working in large companies makes me think it would still be unlikely. "Not my problem" is necessary armor at a company with 15 levels of management between ICs and the C-suite and a rich weft of (official and unofficial) "dotted lines", where every interaction outside your immediate team is fraught with political implications. Code monkeys, in-house or not, might not even know that they are working on software for an airplane, let alone a Boeing airplane, let alone in a safety-sensitive system. Even if they do, and understand that the overall plan they are contributing to is folly, there is often no linkage by which they can make that insight known to someone who can change things (often there is literally no one, including the CEO, who could change course even if they want to). Even trying to do so is often a career risk.
Large companies are mechanisms that run on their own. People have limited agency in a lot of situations but for big decisions ("our company's only real product is fundamentally dogshit, lets start over") there literally are no people on earth who can actually make them. They are pre-made by the structure of the company, capital markets, and historical accident.
birthday balloon? In front of a landing plane at 170mph?
In a safety-critical design you have to take in account what will happen if what cannot happen happens.
Yes. It happens. In a recent news clip a pilot was interviewed about the MAX and he raised the risk of bird strike and birthday balloons as something that “happens all the time”.
It actually happens, similarly to bird strikes or collisions with essentially any object that can float or fly into the path of a plane.
https://simpleflying.com/avianca-airbus-a319-balloon-strike/
https://www.latimes.com/archives/la-xpm-1994-09-22-me-41720-...
https://dronedj.com/2021/08/31/update-faa-says-envoy-air-pas...
GP is (quite reasonably) using a deliberately slightly silly example as a rhetorical flourish. I would do the same. Think of it of a stand-in or totem for all of the possibilities (from silly to very real) that leap into horrifying life in an engineer's brain when they learn that a critical system has a single point of failure.
The biggest problem with offshore engineering teams is their dedication to delivering exactly what you ask for. One of the qualities of a good software developer is to take the requirements and interpret what needs to happen then deliver appropriately. Delivering exactly what's written without thinking about the implications is a huge problem and one of the biggest issues I've run into working with offshore teams.
I agree with you on the problem with offshoring, having done this myself in a previous role in a regulated industry.
However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations.
As such, they were (likely) not the ones who specified that MCAS should silently turn itself on after a pilot turned it off. That misjudgment probably lies with the engineering team who made that design decision, and it had tragic results.
> However, the offshore team are usually not intended to act as domain experts. In fact, they were very likely explicitly proscribed from interpreting the specifications handed to them to guard against them trying to act as domain experts and delivering something different from expectations.
Finally! Thank you for stating this explicitly.
In safety critical systems specifications are everything and is always done by a team of domain experts to an enormous amount of detail (with optional formal methods verification). The actual coder has to just use the chosen implementation language carefully to meet the specification with no personal interpretations; everything has to be explicit and documented.
And yet, the specification is not what executes. If the code can be written by a domain expert, why compromise? If nothing else, more time spent working on the problem by a domain expert gives more opportunity to identify deficiencies in the specification, or the ability to meet the specification within the constraints of the actual execution environment.
Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
>more time spent working on the problem by a domain expert gives more opportunity to identify deficiencies in the specification
So much this. This is happening in other industries, too. Domain experts are also becoming less expert because of lack of proper feedback about their designs.
It's great to have a spec, but you should understand what's happening with the spec and how it applies to the actual code that executes.
With software we can get software that is very close to being an executable spec. By adding a layer of indirection we make everything harder.
Developing Safety Critical Systems is nothing like developing "ordinary" systems. The stakes are so much higher that absolute rigour at every step is demanded and enforced. This is why "Software Engineering Process" methodologies were invented and enforced using "Formal Methods". When this rigour is lost you have catastrophic system failure like in the "current-day" Boeing company.
> Never mind that this idea of an offshore team diligently implementing the spec to the letter hand-waves away the software engineering, as if it’s a mere implementation detail not intimately connected to the system delivering the desired safety and performance characteristics.
If the "offshore" team does not have the requisite Domain Expertise (which seems to be the case here), then it is Boeing's job to provide rigorous specifications and more importantly have safety checks/verification/tests/etc. in place to guarantee "correctness to spec." Problems in the specifications itself are the responsibility of the Boeing Design/Engineering team.
The problem is domain experts aren't actually that good specifications either.
Do you know who is trained to be good at spec'ing software? Software engineers.
You have missed the point.
Domain Expertise is an absolute necessity to come up with Rigorous Specifications and in particular; for Safety Critical Systems.
If they can be done by a single person (whatever be his role name) all the more better but usually for Safety Critical (and highly specialized) Systems that is not possible and hence you need two (or more) people.
If just deliver exactly what you are told to, you aren't doing any "engineering" at all.
Right,I just spend a little extra time on prompts and get much better, faster, work done completely stress free.
Coming from one of those countries where a lot of engineering gets offshored to, the best and brightest engineers in those countries DO have options (no, not just immigrating to a rich country) and won't be a code monkey for multinationals.
I have been to IBM offices in São Paulo a few times and let me just say, although they paid reasonably okay salaries it wasn't the first choice of work for most engineers...
And that is on top of the problems you are describing.
Now imagine a constraint written in management legales. "The system has to always augment flight characteristics to avoid system envelope violations. Should the pilot choose to deactivate the system temporarily, the system must return to enhanced flight characteristics as soon as the pilot stops signaling the intent to fly without the system. "
> As I understand it, the software performed exactly as designed
This is very likely correct.
People on HN don't seem to understand how software is created in regulated industries outside the tech bubble.
In a complex system such an aircraft, the behavior of the system is modeled and detailed requirements are generated.
These models are created by the system engineers.
The models and requirements are then handed off to the software engineers to implement the modeled behaviour in code.
The software is then tested to see if the behaviour matches the models.
So it doesn't matter how much the SWEs were paid, if the software met the requirements, and implemented the models as designed, then the software engineers did their jobs.
The software engineers job is also to think about what happens in corner cases.
What if sensor malfunctions?, how should software act?, will it crash the plane?
If yes, then raise problem and refuse to implement / avoid killing people.
Then you have QA which should have the same questions and test plan to make sure corner cases are covered. QA is also responsible.
Then you have management which is supposed to make sure there are people skilled enough doing the above and the initial systems parts and encourage this environment.
Then you have business people to ensure sufficient resources to hire competent people above.
So what is happening is a massive problem everywhere...
> If yes, then raise problem and refuse to implement / avoid killing people.
Often, software engineers are pipped, written up, berated, stack ranked - and churned via various management processes for doing exactly as asked for.
Until the culture of the company aka ALL levels of management changes to full accountability without blaming the actual engineers, nothing will change. The current corporate structure just isn't set up to incentivize good work.
Perhaps software engineers looking for accountability would like the same level of liability that is enjoyed by their peers in civil, mechanical, mining, and electrical engineering?
> Perhaps software engineers looking for accountability would like the same level of liability that is enjoyed by their peers in civil, mechanical, mining, and electrical engineering?
If it comes with the ability to say NO, or DELAYED, without management song and dance, it will get traction.
Problem is software engineers are replaced with juniors/contractors that probably do not know any better and are only one part of a system that has failed in its entirety.
What sane person would willingly put up code that can kill others on purpose... so we're left with incompetence.
Accountability, if implemented, should be company wide.
Right now issues have passed design stage, implementation, qa... with this sort of failure you have management to blame for not putting up processes in place so its caught / hiring and keeping competent enough people.
This is sort of expected as they optimized cost to the detriment of everything else.
In my experience there is a correlation there. Low-cost outsourced code is generally pretty bad.
Not that you intimated this, but I don't know that it's necessarily a problem with the people writing the code. What I've seen is that organizations that try to save money by offshoring are not providing adequate domain knowledge or usable requirements to the developers.
Throw in a massive timezone difference and every misunderstanding winds up living in your system.
Culture issues do exist. IME different cultures have very different approaches to communicating e.g. “I don’t know” or “I made a mistake”. Not everyone in a particular culture acts the same, but it definitely affects the average outcome.
It’s not a problem with individual offshore engineers. The problem is the way the business they are employed by operate.
That could be. However I think the language barrier is also a significant factor. But of course there is nothing inherently problematic about offshore developers.
I've had to deal with offshores my entire career and have even flown out to train them up. They are good people, but almost universally, they contract for U.S. companies via large body shops (you know who they are) that have awful pay, poor treatment of developers, and just a general culture of half-assing work. I've told various great offshore developers personally to leave as fast as possible and join literally any other organization for way more pay. Good developers have zero business working at those body shops that all the U.S. corps utilize.
My experience is false? Ok..
Correlation is the inference that you draw from your experience which can absolutely be wrong.
Correlation is the inference I draw from my experience? That's one of the more nonsensical things I've ever heard.
Perhaps a course in "Inferential Statistics/Statistical Inference" is in order here.
I've taken several graduate courses in statistics. That's how I know this makes no sense at all.
> several graduate courses in statistics.
Ha, Ha, Ha!
Experienced engineers don’t just build things to spec. They question the spec and the human experience of using the technology.
Yep, the problem with the software wasn't the implementation, it was the spec: to operate contrary to the expectations and experience of pilots.
The problem was that they didn't outsource the design spec. I doubt an externally developed design would be so contrary, with a flippant we'll document it and include it in the short brush up training.
The umpteenth comment trilling on as though MCAS was built by contractors when it was explicitly stated in the article that it was developed in-house by Boeing employees.
Still blame the ones defining the spec rather than implementing it.
MCAS wasn't outsourced to India LOL. People here are gullible idiots hahaha.
Yeah would almost certainly be ITAR controlled no way core tech like this gets outsourced.
What's the difference between a company in India, and a US company full of H1-B workers from India?
And yet we seem to conflate salaries with competent leadership positions.
I think you are comparing apples and oranges here.
An individual contributor is typically given a tangible task (eg: deliver a product that does x in y months for a cost of z). There are several qualified people they could choose from to achieve this task, which drives down the cost. I worked for a company with 10,000 engineers. Oversimplifying for effect, but their skills are not unique.
A leader is given a far more intangible task (eg: gain 30% market share in z years for your division OR deliver 9M of product per day through 202X), and they are taking a huge risk. The team under them may not have the slightest clue how to do this. The interest rates can change. Currency fluctuates. They are paid to calmly chart a path through the chaos and deliver a result. At this level, there were about 60 of us, and the atmosphere gets rarer as you rise further in the ranks. They made absolutely certain we were kept super happy because we could create tangible value orders of magnitude above the average employee, and they REALLY did not want us working for a competitor.
Is this more, or less, concerning than the growing trend to outsource software to $0.03 / 1000 token AIs?
Don't worry, with chatgpt the software is going to be safer. The plane won't even take off :)
"To save us time I tasked my AI, Anton, to debug the code, and Anton decided that the best way to get rid of all SW bugs was to delete the entire codebase, which is technically and statistically the right solution to what I tasked it to do."
-Gilfoyle
That's why we shouldn't ask AI to fix climate change
Fivrrr for airplane safety,… it’s like that BM episode, flywheel of death…
> BM episode, flywheel of death
what is this? do you have a link?
They are referencing Black Mirror but a cursory search with those keywords does not return an episode. I'm sure with a bit more searching that you can find it!
AI generated black mirror titles? Seems fitting
I am not surprised. When in doubt and when you have screwed things so badly, blame the people who don't look or speak or earn like you. Boeing did have $9 software engineers working for them. They were working on the accounting and ERP softwares. But as soon as 700 people lost their lives, apparently they were working on MCAS, a core system that only NASA engineers should have worked. I dont believe anything that Boeing says or does, the company is pure evil. If it's Boeing, I'm not going.
I think the big question is how did no FAA or whatever inspection show all those safety implications until now? What are they missing at other Aircraft manufacturers?
What other aircraft manufacturers? Cessna?
Man these Indian bpo companies like Wipro Infosys HCl destroyed the future of India and gave the world the worst software ever.
> The Max became Boeing’s top seller soon after it was offered in 2011.
Seriously? I mean, could you still order anything but a Max from Boeing in that size? And that being the most useful size, of course it became the top seller. Because it was the only offering.
Outsourcing is a very bad practice. Those softwares tend to be very low in quality.
Wait til Devin makes your next safety critical system. :)