The biggest issue isn't the software glitch system. It's the legal system that threatened innocent people with prison for theft unless they admit they are guilty for crimes they didn't do and pay for damages they didn't do.
This case was brought to public attention and repairs were attempted only because it's huge and involved hundreds or thousands of people.
How many disparate cases there are, where people's lives are destroyed and innocents are rotting in jails, we have to ask?
It's correct that this is ultimately a failure of the legal system.
However, the role of software here must not be minimized. Software makes it easier than ever to diffuse responsibility and create opaque processes that leave the least powerful people at the bottom of the hierarchy holding the bag. By rigidly encoding flawed assumptions and executing them without question, software is the ultimate realizer of our Kafkaesque nightmares.
> However, the role of software here must not be minimized.
No argument that the software bears fault, too.
However, when you accuse someone of stealing money, you should have to prove that they stole the money. This isn't some invisible crime. There should need to be evidence that the stolen money went into their account, got spent to buy something, got pulled from the till on camera, got transferred to Bitcoin--something.
The fact that all these people got convicted with no evidence that the money was ever in their possession is a gigantic legal problem.
It's not entirely unlike Therac-25, including the deaths (albeit more indirectly caused in this case.). There was a certain element of operator error, but that doesn't excuse the faulty programming.
It also doesn't help that the lawyers in the case fabricated evidence and covered up the issues with Horizon to secure the convictions. The people doing this need to spend some time in one of the overcrowded prison cells they sent the sub-postmasters to.
I think for the courts the issue is a bit more subtle. The question is, who's job is it to prove that the other person is wrong ("burden of proof")? Should it be the job of the prosecutor to prove that Intel's processor produces the right answer when an ADD instruction is executed? Or should it be the job of the defendant to show that Intel's processor doesn't produce the right answer? What about proving that the compiler produced binaries which faithfully represent the algorithm? What about Excel?
In our normal life, if a computer is doing the wrong thing, we don't start by assuming a broken compiler; we start by assuming that the new, not-well-tested code is probably broken.
It seems that in the UK before the 90's, the burden of proof was always on the prosecutor to prove almost everything about the system, which is kind of ridiculous. So they passed a law trying to fix it, but messed it up the other way, putting the entire burden of proof on the defendant, without giving them any real way to disprove it. (I mean, shouldn't "discovery" at least mean I can inspect the source code?)
A more balanced law would say that widely-used software with extensive test suites can generally be assumed to be working properly; but that custom-purpose software needs at least some level of evidence that it's correct, and that defendants have a right to inspect any software that's used against them in court for defects.
And that skillful programmer will fight with all power to avoid any kind of minimum standard and liability for crap software, continuing the cycle and abuse.
It's always spectrum from THE SOFTWARE IS PROVIDED “AS IS” to high-assurance methods used in aerospace and similar safety-critical fields.
The skillfull programmer may accept liability when you give him a verification team with a few PhDs, the ability to withhold signoffs, flexible deadlines etc. etc.
Few are willing or required to pay for that. So they get a mystery box with a 90% chance of crap.
This is a somewhat Sith-like dealing with absolutes.
Laypeople generally understand that software may crap the bed in the sense of "the system is down, please wait, then try again". But few people have experienced subtle changes in stored data.
A judge looking into his document cloud may be ready to see a "sorry, not available right now" notice, but doesn't expect that some sinister program is, in the background, silently editing texts of his judgments and pronouncing people guilty when he intended to free them etc.
The problem with the Horizon scandal is in this sinister manipulation of data. It may also have been done by Fujitsu people themselves, in order to cover some tracks and tamper with evidence. This is a very untypical failure mode.
A big problem is they don't want to do the work to understand that, which is the exact outlook the PO had...
"We need some software, ok let's get a big reputable company in to do it for us, we shouldn't get bogged down with all those horrible technical details"
Not wanting to defend the PO but it wasn't really their decision - it was a PFI (private finance initiative) foisted upon them by the Tory government of the day as one of their recurring "STOP BENEFIT FRAUD!" lunacies.
There are not as many that involve jail, but there are a variety that involve ruined lives and even bankruptcy. A recent example is the Phoenix pay system that was used to pay Canadian federal government employees and contractors. https://en.wikipedia.org/wiki/Phoenix_pay_system And I agree that the problem wasn't the glitches. I personally think it was the corporate governance that failed, not the software development and debugging process. The legal system was complicit and enlarged the overall consequences, but the but for test tells me that it was the poor corporate governance that was at fault for a root cause.
Exactly, and by the number of errors among the death-row/lifelong convictions - which one would presume are the most sensitive and carefully reviewed ones - these numbers are HUGE, especially in the US.
> The biggest issue isn't the software glitch system.
I disagree, the software glitch was the problem here.
We are supposed to be able to rely on computers to store and add numbers or report a system failure. This accounting software showed in black and white that some funds that the sub-postmasters were responsible for had gone missing.
What else was the legal system supposed to do? The broken software was simulating crime perfectly.
Absolutely wrong. Mistakes happen. Bugs, fat fingers, laziness, hangovers--whether by human or machine, errors occur. The legal system was supposed to uncover the facts. Because of the Post Office coverup, the judges were told "
no, there are no bugs. No, nobody has remote access to these terminals. Yes, the only possible way these figures could turn up is through theft." This despite the fact that at least one Post Office inspector explicitly wrote in a report that there was no evidence of theft. The legal system failed to penetrate the veil of lies and find the truth. That's a legal systemic failure.
It wasn't though. If the post office enforcers had taken even a cursory look at the transactions around the 'thefts', they would have noticed obvious errors. One of the bugs basically just duplicated a close-of-day transaction, sometimes many times. This would obviously have looked like an error, it would be a stupid way to commit fraud. It was obvious that the Post Office just preferred to extort money out of the postmasters as opposed to actually work out what was going on (as evidenced by the bonuses for successful payments or convictions)
> I disagree, the software glitch was the problem here
Except it wasn't; the main problem was how the PO was handling it. ICL/Fujitsu were aware of near-identical bugs in an earlier project[1], and PO employees omitted parts of an audit from 2004 that described similar issues as well[2]
It all goes back to ICL/Fujitsu and the PO being aware of the issue and withholding the information from anyone not already "in the know"; lawyers, judges, changing witness statements to hide incriminating evidence, etc.
I think if a handful of people had been prosecuted then it would still be an outrage but understandable. But this was hundreds of cases. I think the legal system has some responsibility for not maybe thinking "Huh, what are the chances of so many previously law abiding people all committing the same crime in the same time period?".
This case was brought to public attention and repairs were attempted only because it's huge and involved hundreds or thousands of people.
How many disparate cases there are, where people's lives are destroyed and innocents are rotting in jails, we have to ask?