Rather than the Therac-25 debacle, I would more recommend looking into the Toyota 'unintended acceleration' case. And the legal fallout from that. Because it is terrifying. Toyota was essentially as grossly negligent as it is possible to be. And the result? The court said there existed no standards they could be held legally liable for violating. So your self driving car? It will be developed by junior developers hired as cheaply as possible, driven like slaves by business-oriented managers who only care about meeting schedules, not given the tools or information needed to do an effective job, with testing time cut short and any claimed 'industry standards' for safe coding ignored. The automotive industry had 90+ coding practices they list as either 'required' or 'recommended'. Toyota followed 4 of those in their code. And the court said this was OK. Do you think Toyota spent tens or hundreds of millions of dollars rebuilding their entire development infrastructure, hiring more competent software engineers, firing the business managers who got people killed by rushing an unsafe product to market, and putting the engineers in charge of all future decisions regarding scheduling and release? No, of course not. If anything, they probably saw it as carte blanche to make things worse.
Wasn’t the cause of the “unintended acceleration issue” hardware and driver error related rather than software related? That’s what I’ve read in one book about Toyota management (the Toyota Way) and also a Wikipedia entry on the topic says:
> Toyota also claimed that no defects existed and that the electronic control systems within the vehicles were unable to fail in a way that would result in an acceleration surge. More investigations were made but were unsuccessful in finding any defect until April 2008, when it was discovered that the driver side trim on a 2004 Toyota Sienna could come loose and prevent the accelerator pedal from returning to its fully closed position.[4]
Based on those two sources it seems the issue was hardware related, and Toyota may have tried papering over the matt issue. The faulty matt design issue doesn’t support your claim of shoddy software practices and hiring underpaid junior developers. That may still be the case but it appears not to have caused the SA issue.
This doesn't change anything GP said — the Denso/Toyota code running in the ECU is literally impossible to test or review. Safety mechanisms and watchdogs are faulty and not operational.
This is not the kind of code that should ever have full authority over a 250 kW drive system, even less so with humans in and around that system.
There was a hardware issue, Toyota told their engineers one of the boards would have ECC RAM, but went cheap and used non-error-correcting RAM, but that wasn't the primary issue. After the court case ended, the code was subjected to static analysis by various researchers and a litany of problems were immediately found. Race conditions, uninitialized values, lack of fail-safe structure, spaghetti code, etc. I read a much better summary article awhile back that talked about automotive industry standards for embedded design and the working environment at Toyota (where developers had no static analysis tools, and didn't even have access to a bug tracker), but this article covers some of the points. The software was definitely at fault in many of the cases that killed 38 people:
That article makes a clearer picture of the issues you were hitting upon. Especially the later software audits. Perhaps you could add it to the Wikipedia page?
Because proper design for a safety-critical process shouldn't assume anything. Shouldn't assume that an accelerator pedal will have full range of motion when attempting to disengage, and should be programmed to route around unexpected inputs (or lack thereof)
This isn't reasonable. A traditional mechanically linked accelerator pedal has the same failure condition. If the pedal becomes stuck during a journey, I don't see how you can distinguish between that and user intentionally holding the pedal steady. Having the brakes override the accelerator is a reasonable safety advance that's more possible with direct computer control of the throttle, however.
I think the problem was that the software didn't take into account the situation where the inputs were both the accelerator floored, and braked floored - it should of course result in "disregard the accelerator input, do brake". The user/driver would have to fully disengage the brake, then re-apply it, for the sw to actually brake. (if I remember correctly)
