What really bugs me about all this: not only were they designed to break down, they messed that up to the point that the trains could have broken down while in service. The fact that a manufacturer would risk the lives of the passengers of the trains should result in personal liability for all of the execs of that company.
When bad things happen, the execs don't know it happened because they tell other people to go off and do stuff. When good things happen, execs take all the credit because the told people to go off and do stuff.
When my wife was in business school, the teacher of her management class used this idea as an unironic definition of management. They had to memorize the definition, which was essentially "taking credit for the work of other people." Not as a joke.
The issue started back when highways were being built across the US. Subsequently the US deregulated the Rail industry so that it could compete with long-haul trucking. Its been a steady spiral since then. John Oliver did an amazing and informative piece on trains.
Since then, workers get no days off, they reduced the amount of workers on a train to 2 and are trying to get it down to 1 as we speak. They increased the length of trans by literal miles, and these trains have killed multiple people by blocking roadways / ambulances and have run over children who have had to crawl underneath the trains to get to school. This happens multiple times a week where the road is blocked for almost a day.
These conductors have to walk the length of 4 miles to "inspect" the train, and deregulation has pushed the inspection time down to mere minutes instead of an entire checklist. They have taken all the power from the single regulation body and allow trains to run with brakes built in the early 1900s. We have many multiple "accidents" a year with chemical spills, all for profit seeking behavior. These workers are consistently over-worked and cannot take any sick days unless they are scheduled 2-3 months in advance.
Train accidents are inevitable at this point, the next one may be more disastrous than Palestine Ohio. This is the direct result to profit seeking by corporations (who pull in billions a year) and deregulation. It sucks because trains are f*cking awesome and could help society in a million ways.
Wrong. Trains generally stopping at intersections already pose a real risk by blocking emergency vehicles, resulting in cases of loss of property and life that could otherwise have been avoided. Having an unplanned shutdown that cannot be easily remedied only further exacerbates that potential.
> It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
Yes, but: it would be much harder to test whether bricking this thing selectively does what it should do and for all we know right now you'd have a runaway on your hands. So this isn't just for shits and giggles.
And even a stopped train on a live track can under the right circumstances be extremely risky.
You wrote that the passengers' lives were at risk.
This is simply not the case. It is a fundamental feature of every signaling system, and has been for over a century. Any collision would be caused by a seriously defective signalling system, not the stopped train.
You don't stop a train on a live track without a reason and yes, it would take multiple faults but since we have seen several such accidents in the last couple of years alone it would seem to me that it is indeed a risk. Poland has a ton of aging infrastructure, I wouldn't make any assumptions about what works and what doesn't, systems fail, and trains fail, when both fail at the same time your chances of an accident go up quite a bit so you try hard to make sure that both don't fail. Making one of them fail on purpose is a really bad idea.
Anyway, I'm sure you'll find a new reason to say why it's perfectly ok to stop trains with passengers in them willy nilly and how that isn't a safety issue but I'm just going to let it go here. I certainly hope you're not in charge of anything that involves public transport.
So I’m an engineer in the passenger rail industry who often reviews and has input into the HTL (hazard tracking log). Stopping the train is assumed 100% safe for passengers and crew.
(Edit: you might be able to find some of these online, as regulators sometimes publish them for comment if there is a waiver request… not going to dox myself though)
Your point about external grade crossing hazards is valid… but also completely avoidable unless the train is super long (IE: not a passenger train) or the train was already going slow (like just leaving a station or signal).
There are separate system safety plans to ensure that timely/safe evacuation is possible regardless of where the train stops.
In the absolutely worst case, this means coupling another train and waiting the ~30 mins for people to walk through the train to the new cars, then uncoupling.
You're an engineer in the passenger rail industry in Poland? If not then you probably will want to familiarize yourself with the state of Polish rail infra before commenting further. It's a small miracle they don't have more accidents than they already do.
There are only two countries in the EU that have worse rail infra than Poland (Romania, Bulgaria). Rolling stock is off mixed quality and vintage, maintenance spotty at best.
What makes a stopped train on a track 100% safe in terms of other trains or locomotives that might be oncoming? Are signals and means of communication redundant in a way that there‘s no chance of a collision?
The signalling system is designed so there's no possibility of an oncoming (or path-crossing) train. On a modern system this is done with formal proof of the software, on an old system with mechanical interlocks.
Yes, and if you maintain it very well you will never have accidents. That's why Poland never has railway accidents, perfect maintenance and everything 'just works' /s.
It's like the code was designed to get my wife on my case after, "Honey, don't worry, I can fix it myself and save a bundle not having to go to the dealership."
Uh, maybe not. These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
> These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
This makes little sense. It's pretty reasonable that any machine that is sent for repair may take longer than expected in the workshop. Parts availability for one. Also, manpower shortages, scheduling (we don't need it back until next month) and so on all make this "heuristic" more likely be a scam. This idea that the manufacturer is entitled to a lifetime stream of repair revenue has to stop.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
The top comments on that nypost article are insane, blaming the victim for probably doing "unauthorized repairs", and saying that the "southern border" is at fault.
I feel like blaming guns might upset some people, but I don't feel like this type of thing happens as much in the rest of the western world, where gun laws are typically a lot stricter.
I assume most of it is access to guns. But I do wonder how much is due to people are not using giant ego-soothing trucks in the parts of the world where petrol doesn't get subsidies. So there are fewer truck owners to do the murders.
Ah ok, I see now. Yes, between deranged and entitled customers and the ready availability of point-and-click interfaces of the fatal variety you can have some pretty bad outcomes.
(1) This was scheduled maintenance service, not a quick repair. Maintenance is complicated and is expected to take more than 10 days:
> Maintenance a train is a complicated affair – it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested. The SPS carries out the maintenance procedures according to the relevant maintenance manual (some 20,000 pages) provided by the manufacturer, but the train does not start after being put together.
From [1].
(2) If this was a legitimate check then there should be a legible error code instead of the train randomly
locking up with no explanation why. This was clearly designed to sabotage competing maintenance service companies.
The manufacture's current position is someone hacked their systems and changed the code so that it would make them a lot of money, they totally didn't do that.
>"The president of Newag contacted me," Cieszyński wrote. "He claims that Newag fell victim to cybercriminals and it was not an intentional action by the company. The analysis I saw indicated something else, but for the sake of clarity, I will write about everything.
>Newag president Zbigniew Konieczek said that "no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software."
Which is obviously bullshit made up by the PR agency they hired, designed to muddy the waters. For example, researchers dumped the Train's firmware (in presence of third party witnesses) before and after sending it to the Newag's service, and found that the a backdoor code was recompiled and updated there.
> These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
I know HN is a great place to practice PR-speak, but could you please stop fucking with the gullible people?
The words were chosen to indicate uncertainty on my part. A train parked in a repair depot for a fortnight triggering a need for more checks before putting it in service? I can totally imagine that spec being signed off in meetings. Fortunately the voting system on here does an excellent job of hiding unpopular thoughts so I doubt I've confused anyone.
You’re just pushing a pretty blatantly uninformed opinion supporting a frankly indefensible position. My car had minor collision damage and it took the manufacture’s 1st party service center 3 weeks to fix it. The assumption that manufacturers are uniquely positioned to repair their products is a poisonous tactic being used by insidious players to discredit right to repair.
This is stretching my sarcasm meter to the breaking point, either you are very good at deadpanning or you are trying really hard to muddy waters that are crystal clear. If the former, well played.
Of course, because we all know that only the manufacturer of such a complicated device can be trusted to maintain it. Oh wait, maintenance of trains is a competitive process? The manufacturer provides a 20,000 page maintenance specific manual? Maybe trains really are simpler than iPhones.
/s
Parent comment is really an indication that society has lost the plot when it comes to ownership. The trains do not belong to the manufacturer after sale. They can't introduce anti-competitive code and pretend that users want it, like phone companies can.
That would make sense if we were talking about something that changes dramatically over time; like a human heart for transplants. Or a nuclear fission core without its carbon bars. But trains don’t change significantly in a 10-day period. I doubt they degrade much after a full year, even. They need to be sturdy.
Ok so no sarcasm, well, there are many reasons a train can spend a long time in the workshop:
Complete checkup takes time, or one important mechanic is sick, delaying things or whatever. It is not the responsibility of the manufactor anymore. (A different company has the official service contract.) If you have other informations pls share.
The OP article is worth reading for the brazen blustering statements made by Newag management, I paraphrase "you can't prove that the software found on our trains that sabotages our competition was installed by us!"
That headline was so misleading. I'm interested in the topic and saw that link on HN, but scrolled away because dieselgate is "obviously" about emissions and I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
They should have mentioned something about vendor lock or right to repair in the headline.
> we all know already they are big polluters but we kinda have to deal with it.
There is a lot of variation in train emissions, but most modern ones (electrified) have minimal emissions - and far lower than electric cars even, per passenger mile or per tonnage transported.
Even the oldest ones in common use (diesel electric) are more efficient per ton-mile and passenger mile than a typical car.
> I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
In absolute values, not relative per ton of cargo or compared to other modes of transport. Maybe I wasn't clear enough, for that I'm sorry. I do support trains everywhere and think they are the best land transport we have by far. I was merely thinking that a diesel engine is a diesel, even if a train vendor fudge the emissions somehow (hence the reference to the Volkswagen dieselgate, I thought it meant) it wouldn't matter much on the planetary scale. Thus not particularly interesting.
But instead this was all about electronics and malicious locks, which wasn't apparent from the headline.
IMO this is really what Dieselgate was about. Not specifically fudging the emissions, but hiding secret code to do so during specific test profiles, then lying about it afterwards. Analogously, these trains hid secret code to disable them after being taken to certain locations.
> ... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
Modern diesel engines yes, but the old ones? They got barely any emission controls, only for locomotives built after the 90s there is regulations [1]. On top of that comes brake dust [2] which is only irrelevant in powered-car electric passenger trains with regenerative braking - every other train releases insane amounts of it, no surprise given that the brakes have power outputs in the megawatt range.
>According to Dragon Sector, Newag entered code into the control systems of Impuls trains to stop them from operating if a GPS tracker indicated that the train was parked for several days at an independent repair shop.
Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
“Newag president Zbigniew Konieczek said that ‘no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software.’”
The ridiculousness of this defence makes it clear leadership was in the know.
> The ridiculousness of this defence makes it clear leadership was in the know.
https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne...
- this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.
So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.
IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.
And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.
The 2021-11-?? date was the day the train was scheduled to be serviced (so the intention of the code was to brick the code after the service). Only by random chance it didn't work, because the train was in maintenance a bit longer, and the code didn't trigger in January 2022 because of the date checking bug.
Riiight?! This is a really strong defense for the white hats. Either it’s your code and you are egregious slime balls doing illegal things, or “some third party interfered and hacked your trains like you said and we fixed it you’re welcome”.
There's also a bunch of service people involved etc. (some "non-working" trains were sent back to NEWAG, and the people doing the maintenance had to know how to unlock them, etc.)
What lone developer says "let me find all the GPS coordinates of all the train repair shops and make the trains not work if they've been parked there for days"?
"My cat slept on the keyboard and accidentally developed and merged working advertisement feature into a release build of the new AC game. It was totally on accident, happens all the time." (c) Ubisoft
"I come to you now with apologies. This is the third time my cat has put paid mods into Skyrim, something for which he is very very sorry you are unhappy about."
We talk on HN a lot about ethics and programming and the importance of engineers refusing to develop unethical software.
But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.
Humor is prevalent in the internets various forums. It's well appreciated generally - easy upvotes on Reddit and elsewhere. But that also means that a serious forum with higher signal-to-noise ratio is something uniquely valuable.
Perhaps GitHub Autopilot and ChatGPT teamed up to write the code upon noticing that the developer tripped and fell, causing a GSM modem to fall off their desk and into the assembly process.
As a lone developer there is absolutely no motivation to do this and hide this decision from management.
Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?
There is just no possibility where upper management is not involved in this.
100% agree. Doing this in secret without management's express approval means no bonus, working extra for no benefit, compromising your other tasks and exposing yourself to heavy liability for compromising the country's infrastructure. This is something that companies could get away with, but a single developers would not only lose their job, but risk going to jail.
Note that management's express approval could be verbal conversations that you cannot prove happened or not. With some work management can find some way to get a developer who has some other conflict of interest to do this: if your brother in law works for the maintenance yard situations then you have motive to ensure he has work.
They will surely try that explanation too if the others fail. Currently their stance seems to be "there is no such code on the trains, and if there is, it's not from us".
That didn't happen, and if it did then it wasn't us, and if it was then it was inadvertent, and if it wasn't then our current leaders didn't know, and if they did then it wasn't illegal, and if it was then we have a different interpretation of that law which is unreasonable anyway and should be ignored and repealed.
Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.
You're talking about industrial PLCs. They're programmed using a-bit-more-fancy Scratch snappy blocks. There is no version control. The firmware contained paths embedded as strings, so we know that firmware for each model and customer was developed in a separate folder on disk. I wouldn't be surprised if they also had .zip files with backups of previous versions.
The IDE for these PLCs actually has VCS integration! It's SVN, but it's still better than nothing.
Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.
There is nothing wrong with SVN, it's just that Git allows for some workflows that are better suited to larger teams and more complex projects. But for your average PLC project with a team of 10 and one binary as the output it should be more than enough.
You likely won't see any 'feature branches' or frequent merges in this kind of environment.
I'm aware of repos with a few million files in it that have been going since 2003 and not a single issue.
Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.
I'm intimately familiar with PLC programming, yes, you can do it the 1980's way but there are also plenty of environments that allow for modern version control.
You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output (text) files. You may not be able to do an easy line-by-line comparison but you will have a commit log with helpful messages.
Look for 'engage in anti-competitive behavior' in the log message ;)
Yeah.. I've just realized that while it's entertaining to watch how it unfolds or predict what can happen next, it's also sad, because pretty much everybody in the rail industry loses..
One of my business partners works for PKP it's very annoying to see this all unfold and in this particular way. Poland has so much potential, these idiots are ruining Polands image in ways that really matter.
But then again, as a Dutch person I have enough issues locally that I can't even complain...
Poland has a very strong technology and mathematics tradition that goes back decades. It's one of the reasons Poland has some strong feelings about their role in the breaking of the Enigma, for the longest time that was played down.
Working in security on the operating side (albeit not in Poland):
No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.
For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).
For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.
If this goes on to criminal charges, then they're about to discover what amazing things a thorough digital forensics analysis can find out from their workstations.
If it was developed anytime after 1990 (probably before) you will find plenty of programmers willing to be expert witnesses and tell the court that the company not having version control is gross incompetence, the only reason a company would do that would be so they can hide evidence of illegal actions. As such the court should impose punitary damages.
Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.
I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.
If an employee did this without authorization, the company should still be liable because I do not think safety regulations exempt manufacture from responsibility of employee actions. I believe the opposite is true, that they must ensure safety despite rogue employees, with reviews, audits and such.
They need to show audits and that the code is obscure enough that it could be missed. I've seen IOCCC type things that do things I wouldn't catch in a code review. (IIRC there is a malicious code contest where the goal is to put backdoors in that others don't see in review, but I can't recall what it is)
You’re thinking of the Underhanded C contest: http://www.underhanded-c.org/. The goal is to write C code which looks completely innocent, but which actually performs some malicious or unwanted behaviour.
In the last year they ran the contest (2016), the goal was to write a program that would compare two sets of measurements for similarity (nominally radiation emission spectra) but fail to give a correct answer when given a particular crafted input.
There's been a decent amount of those in the wild, too. If you're in an environment without unicode auditing you can do all sorts of crazy things without IOCCC levels of code tricks too.
Is this really suggesting some developer, buried under tasks and putting extra hours, thought to implement this feature in his free time for months, slipped it through reviews and scans and pipeline deployments, for... what? Really? I rather prefer to think the comment was sarcasm.
Yes it was sarcasm, since it is common for management to shift the blame on individual engineers in these cases (see for example the Volkswagen emissions scandal and the Boeing MAX crash).
In VW's case there was some reason to believe that at first. There are reasons to test an engine in the lab without emissions controls, and so it is believable that a lazy engineer could put in code for the lab that also triggered in real world conditions.
Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.
Even if they did keep that paperwork, they should be punished alongside the manager who asked them to do it. They knew it was unethical and did it anyway. They should have refused. Risk of being fired is not as big of a problem for a programmer as it is for almost anyone else.
I've got mixed feelings. I would be interested in hearing the whole story first. The "hey, I can program the trains to break if we're not the ones to run maintenance, should I do that?" or the "yeah, I guess I could make the trains break if we're not the ones to run maintenance" are both pretty clearly unethical.
However, the "yeah, sure I can add in a GPS locator module" and the "yeah, I can add analytics that reports when the train is in a maintenance hanger" and the "the catastrophic program halt code module used in cases of extreme failure is located here, but why do you want to know that?" all seem less than unethical.
Theoretically you only need one unethical line of code, so how it got there, I think, is pretty important to know before passing ultimate judgement.
EDIT:
Of course for something like train control software, you really should have a process or at the very least responsible engineers that would notice a middle manager with limited technical skills asking suspicious questions and then pushing up a PR that is self approved.
I would be more than willing to entertain an ethical debate along those lines. Although, like I said, I think it's important to understand the whole story because the specifics really do make a difference.
I don't think that's a good way of thinking about it. Just because programmers might have better job security doesn't mean other motivators for not wanting to get fired exist.
Which also applies to everyone else. The programmer job security just makes that easier in most cases.
Which is not really my point as to why they should be held responsible. The reason is the real world consequences of their actions and the scale and ease of introducing negative consequences by tech creators.
I disagree with the framing "You are a bad person" (although I think I understand the sentiment), because it implies they can't change (or understand the error). It seems better to leave it at "You did something very harmful, destructive for society".
I hope they will testify so higher-level managers will be fined and spend some time in prison. And company will be hit with massive contract penalties and fines.
> Newag has said that "any remote intervention" is "virtually impossible."
That’s a very bold statement right there! Also, from my experience, most if not all companies who build automation/robotic systems they maintain “backdoors” of some sort, not to disable them remotely, but to quickly and efficiently access the software remotely upon request to troubleshoot or fix issues, it doesn’t make sense to fly to the client and sometimes the other half of the planet just find out it was a local IP mismatch or a system service is down, so saying “impossible” is not true.
* Polish train maintenance company, SPS, was getting suspicious as trains made by a company, Newag, kept on "randomly" breaking and couldn't be fixed. They was getting fined millions by Polish government as they had a contract that fined them for being too slow with repairs.
* They secretly hired literal hackers (Dragon Sector) for 2 months to dig around Newag train code.
* Hackers found out some incredible things, generally that fit under the umbrella of "late-stage capitalism", or more specifically, corporate protectionism, sabotage, ransom, etc.
Some examples of the secret code that the hackers found:
* Breaks the trains if they go into geo polygons that are right around the warehouses of 5 Polish train maintenance companies, including SPS.
* Breaks the trains after 1 million kilometers.
* Breaks the trains if they don't move for 10 days.
* Secret button press combination (basically Tekken, Street Fighter, etc.) to disable the "malfunctions".
Do you know if the low-level technical report is available? I love reverse engineering firmware, and this sounds like a holy grail. I mean a freaking train? Ugh someone should drop binaries.
Not found one yet. The linked article contains a very small amount of detail, such as the lat-long coordinate values they found within the dissassembled code, etc., but not much else unfortunately.
I'm waiting for the stuxnet-like report on this as much as anyone.
I don't know, is it some kind of requirement to read every piece of news on the main page every single day to keep visiting? Or can I log in every few days, read a few stories, upvote the ones I like, then visit again few days later? Or is that not allowed?
But if you're not around, you miss some stories, that's it. (especially when it's come up so many times over weeks) There is a current events aspect to HN.
I don't read the vast majority of sites linked here. I come here for the comments. I assume anything that's important enough to know will be directly quoted in a comment here.
I was expecting some malicious code like "knock 3 times on this door to enter, otherwise the door locks for good". No. Instead it was "if you're geographically parked at our competitor's lat/long then we'll brick the train".
There's ways to add back doors and other fun easter eggs in code, but wow they picked a really cumbersome and obvious method of exploitation.
Reading their official statement it's hard to not get the impression that it wasn't even consulted with their attorney - they deny specific allegations in a way that just begs further investigation.
I mean, I'm Polish and worked for such people in the past. The statement reeks of an approach to business that I wish went extinct already, but apparently you still see pockets of it here and there.
pretty sure even if they have source code version control, the "funny" patches are outside of the version control and applied on top before making a build.
That's almost certainly not the case. In general for vehicles, you're installing signed, verified images. Not only would it be a LOT harder to consistently apply OOVC patches that don't show up anywhere, having signed builds that don't match your source would be a worse red flag than just having the code in version control.
You do get deniability though. Given their line is "this is malicious code that was put on the train. Physical security is the responsibility of the train companies”, they would argue the fact their binaries are different is proof of malicious actors.
Essentially, if they're going to do this, a good route is for them to be the malicious actors. Just before shipping, someone trusted surreptitiously switches the binaries to the modded versions.
yes, they probably have a "clean" build and "a build with extra functionality" build.
So, they could even install a clean build for some trains.
Though, investigators could just compare firmware from all trains made so far. Another question if it would be even be investigated.
I have seen developers lock/encrypt their code they write and then when they get fired for it, they want the company to pay a fee to get them to unlock it/release source code
We lament that circumventing DRM is legally perilous, but sabotaging your customer's property is illegal. That you were the one that sold that property makes no difference - you sold it, it's not yours anymore, despite corporate messaging otherwise.
Not yet, but when they do, each railroad tie (railway sleeper) will have a serial number printed on it and can only be replaced with genuine Apple® railroad ties.
If they did, the software would be fine. They'd just be held together with adhesives to make repair far harder than necessarily. And nobody would be able to sell spare parts.
This is 100% going to be the opposite case. The question is how, as a developer asked to do this kind of thing, you can defend against the subsequent inquiries.
(remember VW trying to blame two engineers for the "defeat device"?)
Why the assumption that this was some rogue developer? What would they have to gain? I am 110% convinced this was deliberate strategy from the company and not just a random malice from an employee.
Lone pawn, as in doing directly for the strong person at the top (owners family?) what the top would never dare to order through the regular chain of command would not be quite as impossible.
I think the above is actually well crafted sarcasm. But it does highlight why as an engineer/developer it is important to get requirements in writing and have a process that shows why you made the changes you ultimately made.
Seriously? I find it very unlikely this is the work of a "lone wolf developer". Newag management and executives knew about this. Some of these trains were configured (poorly) to "break" at a scheduled date that just so happen to line up with when those trains were scheduled to go get serviced[0]. Lone wolf developers usually don't have the ability to make builds/configurations for a particular deployment of the software...
Seems like you're recommending a system of scapegoating. It would be easy for executives to suggest such a change in a round-about way to managers (Gee I sure wish it was impossible for other companies to service our trains). The managers then field the suggestion to any programmer they know to be particularly "non political" and amoral. Then when the official investigation kicks off, all the blame is laid on the "lone wolf" programmer (who in fact was not a lone wolf because he was chided on by management and indirectly inspired by the executive) and the company itself is let off the hook?
In these situations the entire company needs to catch the blame, even if the commits can be pinned to one programmer. By all means fine or imprison the programmer, but also charge his boss because he knew or was criminally negligent for not knowing, and ditto the bosses' bosses up to the executives. And fine the hell out of the entire corporation; either they knew this happened and did it deliberately, or they were negligent for allowing something like this to happen. Either way the whole corp needs to pay.
> This goes to show you must have controls at all levels in your code integrity. A lone wolf developer in your organization can and will implement these kinds of things to curry favor with their management. While good intentioned, it’s every executive’s worse nightmare.
WTF is this comment? Hanlons Inverted Anticapitalist Razor clearly says to never attribute to stupidity that which can be more easily explained by greed. Therefore it's more likely management that's at fault here.
Developers are likely to want to reject work like this, but are coerced into it by their employers (who threaten workers with loss of work, various punishments, loss of healthcare, reputation damage in industry if workers do not comply with tasks given to them). This was an ownership/exec level decision. Yes, we should be beware of "lone wolf" employers (hierarchies which allow the decisions of some isolated few to coerce many under them).
Managers could fish around for a mercenary-mentality programmer to do this without coercion. They have information on people's backgrounds, know about other morally dodgy things they may have coded before, and know the general temperament of their programmers. Find the guy who used to work for real scammy adtech companies before yours, who likes to espouse extreme libertarian talking points in the lunchroom, who maybe has a gambling problem or is otherwise looking to get a lot more money and seems like he would probably would steal from you if he found a way. Hint that doing the job will be good for his career or come with bonuses. Now you have a compatriot in crime instead of a coerced victim; this way he's more likely to keep his mouth.
This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
On the other hand, trains that refuse to start don't immediately kill everyone on board, and trains that have fall off the tracks do tend to kill people. So when a train fails and people are dead, and people go in search of who is responsible, it seems decently likely that ambiguity between the third party repair shop and the company responsible for the train is a problem.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There should be an opt out system - something where the train operator contacts the train supplier and says "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Or to bring it closer to home, say one of us breaks into our tesla to exercise our God given right to change the software stack, and then it kills the driver and various people around it, to what extent is that Tesla's fault? Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
You are putting forth some general rules that seem reasonable, but they don't really apply in this situation.
From what I've read, the train operator bid out the maintenance for these trains, and the manufacturer lost the bid to a third party. So your "opt-out" system should absolutely have been triggered here.
If there was some reason to believe that the low bidder would be incapable of safely performing the maintenance, the train manufacturer should have raised that issue publicly, not silently sabotaged the low bidder's ability to perform the maintenance.
Maybe? I'm speculating here as it's not my field. The obvious explanation to me is very different to the obvious explanation to every other comment I've seen so far. That difference seemed interesting enough to be worth posting.
The question of where liability falls after a repair isn't covered in the articles linked unfortunately. It seems plausible that it was initially whoever wrote the software and is now someone else.
I think it's extremely likely that this disable-train-on-various-conditions behaviour is exactly as specified and documented since train software is a bit obsessed with formal methods and documentation. It's then only "silent sabotage" to the extent that said docs were ignored.
It's credible that a way to easily disable these safety checks for the case where it's now someone else's liability wasn't considered worth paying for by whoever bought the train. It's tomorrow's problem after all. Also the customer may not be totally convinced of the necessity of the software dev paranoia, especially if they're not liable for failures.
So sure, maybe this is evil/negligent software people at the train company. I don't see that conclusion well supported by the article.
Your argument is based on an incorrect understanding of liability, and is also nullified by the fact that these measures, allegedly justified on safety grounds, were not just undocumented (see other sources) but clearly deliberately hidden, to the point where the manufacturer pretended not to know why the trains would not start.
Update: here's a quote from the Ars Tech. article itself, showing a) no tampering with the trains' software or hardware was necessary to get them running, and b) there's no even slightly plausible case for believing the manufacturer was unaware of this. Furthermore, if it really was somehow unaware of what was going on, then these 'features' cannot be justified as a safety measure.
Dragon Sector got the trains running after discovering "an undocumented ‘unlock code’ which you could enter from the train driver’s panel which magically fixed the issue."
Update 2: It is barely plausible that the ability to track the trains' presence in other maintenance shops was initially added to gather evidence for liability and warranty purposes, and then someone had the dumb idea of using this to covertly disable trains when this happened.
Note that the use of a third-party maintenance operation was not a secret: the train operator solicited bids for the work, and the manufacturer tendered one. Clearly, third-party maintenance was not in violation of any contract (and if somehow it was, the manufacturer did not need to gather any covertly-acquired evidence for a breach-of-contract suit.)
>It's then only "silent sabotage" to the extent that said docs were ignored.
From what I've read, the maintenance company did scour all documentation and found no explanation for why the train was disabled. It was an entirely undocumented feature. Maybe they were just lying, but this idea you have in your head that it was just a fly-by-night operation who couldn't be bothered to read the documentation is contradicted by the available public record.
So all of this obviously biased and incorrect speculation on your part is based on.... not being an expert in the field... and not having read the actual article? Thanks man, very insightful.
- there's a proper registry of who is responsible for maintenance of the train, including chain of liability. Crucially, the vendor wasn't in it for the affected trains - they'd only be liable if design issues were found.
- maintenance companies was appropriately certified and verified - we're talking professional MRO, not random workshop
- owners of the trains were also registered as responsible parties for maintenance and repair
- vendor was supposed to provide documentation that would allow such a workshop to perform maintenance and repairs up to P5 (largest scale) maintenance
- vendor never disclosed lockouts they implemented, thus committing fraud at the very least.
Sounds reasonable to me. Plus you've used domain specific terminology which suggests you know more about this than I do. Thanks for the context.
I definitely haven't gone looking for the documentation on the train system, partly because I assume it's unavailable to the public and mostly because I assume it's enormous in extent.
The same viewpoint applies to any motor vehicle too. How many road accidents are due to mechanical error introduced by a mechanic (that does not operate on behalf of the car manufacturer)? What makes an "official" mechanic employed by the manufacturer superior to a third party?
This is an interesting discussion, thank you for your nuanced opinion. I don't know where I stand on it but you are spot on about the demographics of HN. I certainly have a knee-jerk reaction in the direction of operator freedom.
Edit: Additionally, if something like this goes wrong (say the train derails) it will be a massive blow to right to repair. Even if it is later found that the third party repair is not responsible, the media will scoop up this story and it will forever warp the public's ideas about right to repair.
That's what we have regulations for. If the shops follow the regulations and have the appropriate certifications we know that the work they did is up to spec. Also, obviously in case of some problem the inspectors will check for the root cause and if the repair shop is established as the reason they are liable, not the train company. This is all established law since .. ages.
This is just the next episode in an age old battle. Old cars were easy to repair, so people often went to independent repair shops. Companies obviously didn't like this, so they started using proprietary screws. Then laws were made to forbid this. Next round came when cars got computers and companies didn't give the repair shops the required software and/or hardware to work with them. So, laws got made which forced them to. And so on. Each time there's some new technology companies will try to find ways to misuse it to the detriment of customers. And each time law makers will have to get involved and put the screws on them. And then people later go around "why do we have so many laws" .. because companies are shit, if they can get away with it. It's their natural impulse.
Regulations don't work. There needs to be legal liability for the fraud perpetrated by the vendor against the customer, not prescriptive rules that vendors influence, then follow dogmatically so they can cite compliance in order to disclaim liability for their intended bad outcomes.
Statute and common law are the basis for legal liability. The modern concept of regulation, as promulgated via various administrative agencies, is distinct from, and often displaces, the traditional function of law.
In theory, I agree with your viewpoint. Yes, obviously, safety critical systems need to remain safe, and a workshop that isn't capable of performing maintenance to the degree mandated by the train software shouldn't be doing it at all.
That is not the case here.
If that was indeed the case, one would expect the train's diagnostic systems to report that it has detected certain issues that need to be resolved before the train will start up again. But that didn't happen, instead the train simply refused to start, no error codes, nada.
Not only that, but further investigation revealed that the train's firmware contained code that would disable the train if it was present in a competitors shop for a considerable time.
> There should be an opt out system
This mechanism for disabling the trains, was never mentioned in the discussions or contracts for purchasing the train (which is why a third-party could win the repair contract). There was never an option presented to "opt out", that is why people are saying that the trains were sabotaged.
> to what extent is that Tesla's fault?
It isnt. You made unsafe modifications that can be clearly shown to have caused damage, it's your liability.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Depends, did their software present an actionable error or not? If the car gave an error code, and you look in the manual and it says that too much current is going to the motors (idk), and then you fix that and the car starts? Great!
But if the car just refuses to start without any indication as to what is wrong, and further even if all repairs are reversed it still no longer starts? That sounds like deliberately sabotaging third-party repair.
> This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
Actually, the issue here is that the train company is on the wrong side of this issue.
> Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There are tons of cases where the repairer, not the OEM is held liable. The manufacturer is at fault for the defective parts/product. The repairer is liable for a defective repair. Courts have been sorting this kind of thing out for centuries and it's not a difficult thing to deal with.
> "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Except that really isn't the case at all, and hasn't been the case. If the repairer does not do the repair correctly, the repairer is liable, not the manufacturer. Most mechanical shops even carry insurance and offer their own warranties on their work. Most cities (and hopefully consumers) will only do business with repair shops that are insured for this exact reason.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Yes. It is not their car. It is my car.
Ford is not liable if that variable geometry camshaft I installed in my Mustang causes the engine to blow, killing six people with shrapnel. I would be liable for that. Why should Tesla be liable for someone hacking the software? The issue here is the world doesn't work the way you are describing, where manufacturers are held liable for the work of third parties. Tesla is held liable for their own warranties and laws applying to product safety.
> The code also allegedly bricked the train if "certain components had been replaced without a manufacturer-approved serial number," 404 Media reported.
- https://news.ycombinator.com/item?id=38628635
- https://news.ycombinator.com/item?id=38632033