Agreed.

> and face significant amounts of online abuse from users of the software.

Much as I’d like this to change, I suspect it will not. I’ve been doing open-source work since the mid-90s, and have suffered abuse that would curl your toes, but I still do it, for reasons that would be obscure, for many folks, hereabout.

I think the onus is on the users of the software. It’s currently way too easy to make “LEGO block” software, by splicing together components, written by others, that we do not understand (which, in itself, is not necessarily bad. We’ve been working this way for centuries). I remember reading a report that listed YAML as one of the top programming languages.

If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering.

I don’t think there’s any easy answer, as a lot of the most effective solutions are cultural, as opposed to technical. Yesterday, someone posted a part of an old speech, by Hyman Rickover[0], that speaks to this.

[0] https://news.ycombinator.com/item?id=39889072

This, each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded.

Of course, it will always be hard to detect very sophisticated attacks from actors that can play the long game. But if maintenance is not just a hobby, PRs can receive more attention, and if it's not an added workload besides a day job, there is less churn/burn-out which can be used for malicious actors to take over a project.

Agreed, 100%. And yet: In a comment on Slashdot one guy said that his company had "thousands" of machines running Linux, and he was "proud" to have never paid a penny for it. I called him a parasite: open-source brings an immense value to his company, and he should support it.

My comment got a lot of hate, which I just don't understand. Sure, OSS licenses say you can do whatever you want. However, there is surely some ethical obligation to support something that your entire business depends on?

I don't agree with “usually” there, I think it is an exception rather than the rule.

Where it happens, it tends to be higher profile, top-level, visible, projects that get this treatment from the commercial bodies that rely on them. The smaller projects that they might depend upon are likely to stay less visible, less thanked, and unsupported by those using them (directly or indirectly). What has happened with xzutils is a very good example of this and the potential dangers the situation poses.

I don't know how we should address this. It certainly isn't the responsibility of people like Collin to address, unless of course they want to take on that responsibility.

> if there isn't enough demand or support they simply stop working on it and that's it

Sometimes even if there is enough demand they stop, as is their right, when other priorities come up in their life (or they simply lose interest). A “community” using their stuff does not, and should not, automatically make them beholden to that community.

Honestly the problem feels Like it is capitalism itself … which is weird.

Thus if the license says ‘you don’t have to pay money to use this, nor do you have to do anything at all’ it’s no wonder it gets used for nothing in return. Stallman was obviously right with the GPL keeping the source free instead of end users.

Of course HN is not the place to discuss politics so in some way it is probably for the best that these questions remain unanswered. It would be better still for those muhhh capitalism - and for that matter muhhh whatever-ism - remarks to remain unwritten since they are not supposed to be discussed here anyway.

[1] e.g. https://news.ycombinator.com/item?id=39866101

This is the purpose of licensing and you're describing a failure in the popular licenses that were popularized by the FSF and similar groups.

We have been indoctrinated. It is not always appropriate to give away your source code for free to everyone, and corporations without the moral qualms that individual developers have are happy to take advantage of their naive licensing.

We, as developers, need licenses that are less liberal, such that when corporations use our code, we are properly compensated for the value we have provided to them.

I think this means a deep rethinking of how we license software, and reopening ourselves to the possibility that proprietary licensing is not always bad.

I wrote a little article about that: https://littlegreenviper.com/miscellany/problems-and-solutio...

This is good but I don’t think it’s enough to cover stuff like this which is deep in the stack and probably doesn’t have enough demand to warrant the trouble of direct funding (thinking of how many organizations make it easier to buy a $50k support contract than a $50 open source donation). I’ve been wondering if you could get companies to pay to something like the Linux Foundation, Apache, etc. for a general maintainer group tasked with basically taking something like Debian’s popcon and going down the list of things which aren’t well-established projects to pick up maintenance work.

There could be a lot of variables to the rating, like the size of the team, the language, the testing methodology, etc.

The board would need to be supported by end-users, and big steps would need to be taken to ensure true independence.

It would have to be carefully done, though, and likely immensely unpopular with the HN crowd.

> Size of team

3, with no outside contributions accepted into the core.

> The language

C

Those two, presumably, would end up being heavily weighted towards having a low score.

> Testing methodology

Here they’d probably get some points back based on their ~10:1 test lines vs code ratio.

Just because the kidz doan' like it, does not mean it's bad.

1 is the loneliest number (have to be an old fart to get that), but big teams can be a problem (Camel is a horse designed by committee). I'd say the experience and track record of the principals are a big factor (see "Linux" and "Git").

3 people are easy to evaluate, have been on the project forever, and presumably know what they`re doing.

C shouldn`t be a problem per se -- again, it`s been around a lot, and while it makes it easy to bork pointers, it`s not a meme language that we don`t know much about

This would probably ruffle too many feathers from the GNU old-timers, but I really dont see any other option. We are way past the tinkering-in-the-basement days of Linux/BSD hackers when most of us just wanted a cheap Unix box to play around with or to avoid Windows. A massive percentage of the civilian (and other) infrastructure is built on the shoulders of unpaid hobbyists. There is already massive liability at the social and corporate level. Time to deal with it.

EDIT: Ok, sounds like I have to describe this better: 1) you (governments) force commercial providers to assume liability for security issues and exploits and force disclosure, etc, 2) their insurance premiums go up, 3) to reduce premiums they only use checked/secured software, 4) that means maintainers of at least the critical pieces of software get paid via the (new) channel of risk reduction. Doesnt apply to all OSS, doesnt even apply to all distros. But it creates an audit trail and potentially actual compensation for maintainers.

As for throwing money at the maintainers, honestly, it’s complicated. A lot of people aren’t doing open source work for the money. Money too often comes with strings, requirements to prioritize what the funder wants to prioritize, pressure to perform consistently, it becomes an actual job.

Not only does this turn off a lot of the types of folks who make the best contributions to these projects, but it bends the priorities toward what would make the most money for the funder. And as this article points out, real security investments often fall by the wayside when profit is involved.

So yes, companies should encourage their workers to contribute to these projects, donate money to the foundations that fund them, hire important maintainers and give them carte blanche to work on open source. But we have to be careful. Making it all completely transactional is directly contradictory to what drives a lot of the contributions.

Yes, the devil is in the details, but I think the basic concept is worth exploring

In that particular project, we're all just Paying It Back (not forward), but other projects could likely benefit from the participation of Grumpy Old Farts.

And obviously a non-insured piece of code that assumes no liability whatsoever can still be free and maintained via IRC, same as it ever was. I dont see how this "kills all open source".

This is an attempt to shift costs onto open source developers. Which, aside from being totally unjust, won't work. There's a legal expression "you can't squeeze blood out of a stone". Shifting costs onto people that can't carry those costs doesn't work for the same reason supporting a skyscraper with a toothpick doesn't work. The toothpick breaks, and when the skyscraper lies collapsed on the floor, nobody blames the toothpick. Hell, they might say the toothpick was heroic: trying to save the situation, sacrificing itself, screaming, and when nobody helped, not the government, not the owners, not ... the building collapsed and all the damage was done.

But it's even more stupid than just that. As soon as this gets introduced, and some company makes a security fix. They of course, for GPL or AGPL software, have to release their fixes. This will then make them liable for any other security problems in that same software. After all, they'll be the last ones releasing that software after the government implemented this law.

So how will you even do this, without making software fixes effectively illegal? Achieving the exact opposite of what such a law tries to achieve ... But of course, you can't have this discussion with people just looking to keep "their" free stuff but trying to shift the rest of the costs.

Second HN story from the link above, Dec 2023, https://news.ycombinator.com/item?id=38787005

  The Debian project has completed a general-resolution vote, adopting a statement expressing concern about the Cyber Resilience Act (CRA) pending in the European Union.

  Even if only "commercial activities" are in the scope of CRA, the Free Software community - and as a consequence, everybody - will lose a lot of small projects. CRA will force many small enterprises and most probably all self employed developers out of business because they simply cannot fulfill the requirements imposed by CRA. Debian and other Linux distributions depend on their work. If accepted as it is, CRA will undermine not only an established community but also a thriving market. CRA needs an exemption for small businesses and, at the very least, solo-entrepreneur.

But we (mostly) don't even pay open source developers to write the code ... who is offering to pay them for this insurance?

Besides, this was a highly sophisticated actor. Someone willing to create several layers of loaders and invest long amounts of time into getting xz excluded from certain checks. Anyone with such sophisticated spy craft could have fooled the insurance companies also.

On the other hand, you can already buy software, where the vendor takes some kind of liability: just buy Windows, AIX, or one of the commercial Linux offerings. Same for software libraries: there are commercial offerings that come with (limited) liability from the vendor. It’s even possible to create software to stronger standards. But outside of some very specific industries (aerospace, automotive, nuclear, defense, …) there doesn’t seem to be a market for that.

We’d still be using MS-DOS.

Yes I know the stories of “insurance made steam boilers safer”. And it’s true. But it also stopped innovation in the space before Charles Parsons came along and ignored the whole thing (military industrial aristocracy)

I think the answer sits somewhere in “have less stuff”.

We have millions of lines of code in all walks of life and Inswear we are orders of magnitude over engineered in almost all cases.

If you work for a large company try counting how many different ETL solutions exist, CSV uploaders, data lakes, warehouses and so on

Then imagine having one library to do it.

Somehow we need to get there for … everything

The closest metaphor is cars I think. And yes you can argue that innovation in cars has slowed down but also a 'minimum floor' of safety and efficiency forced by governments and insurers has made new entrants more likely. I.e. you shouldn't need to only trust Oracle, SAP with your business because then, erm, you'd have exactly the current situation in enterprise software...

Ok, I can blow your mind.

You can start your own software projects and offer them with warranty. And people can join you, if they want.

Certainly not for a few € of donations.

Isn't that pretty much the way the world works now? What needs to be fixed?

For the past 10-15 years there has been a strong culture of never writing code when a library on GitHub or NPM already exists, which has, in large part, contributed to this. In many cases using existing battle tested code is the right thing, but it was taken to an extreme where the avoidance of pulling down a bunch of random open source packages with questionable stability and longevity was often maligned as not invented here syndrome.

Now many of those packages are unmaintained, the job hopping SWEs who added them to the codebase are long gone, and the chickens are coming home to roost. The pendulum will probably swing too far in the other direction, but thoughtful companies will find the right balance.

“…comes as-is, without warranties and without any commitment for future work. Complaints will get your feature request deprioritised, may get you banned, and will look silly to any potential employer googling your name”.

Also, let’s make it a meme to call out unreasonable behaviour: stop Jigar-Kumaring!

This wouldn't be entirely without downside though, as there could be a risk that the project ends up getting steered by whoever has the most money, which may be at odds with what the broader community gets from the project. That's difficult to avoid whenever Open Source developers get paid, unfortunately. If it were limited to bug-fixes I think the risk would be slim. I'm not sure if any projects have tried this.

Being pseudo-anonymous filters out alot of credible threats though.

Also, this backdoor was discovered by sheer chance. How many of those backdoors are still to be discovered? We should be checking all source code present in a standard Linux server right now, but all I can see is complacency because this single attempt was caught.

From the article:

  ..the fix for this was already in train before the XZ issue was highlighted, and long before the Github issue. The fix stopped the XZ backdoor into SSH, but hadn’t yet rolled out into a release of systemd.

  I believe there’s a good chance the threat actor realised this, and began rapidly accelerated development and deployment, hence publicly filing bug reports to try to get Ubuntu and such to upgrade XZ, as it was about to spoil several years of work. It also appears this is when they started making mistakes.

Since keeping such backdoor hidden in plain sight is extremely hard and required tons of preparation and social engineering spanning multiple projects, the answer is probably a function of number of those already discovered. As we don't discover years-old similar backdoors every now and then and had discovered this one pretty quickly, this might very well be the very first one that came this far.

Also, what's "sheer chance" for an individual is "enough eyeballs" for a collectivity.

I don't think Andres is in serious danger, unless he is a persistent threat to the bad actors. It's true that we owe him big time for discovering the backdoor. But it could have been someone else before him. And it may be someone else the next time. Too much depends on chance for anyone to justify targeting him. They risk blowing their cover by doing that just to satisfy their ego.

Devin AI working Upwork jobs blew minds, but it succeeded for a reason. Upwork and similar sites are plagued by low quality contractors who do little more than glue together code written by better engineers. It was never a hard formula to copy.

Outsourcing programming work to the lowest cost and quality to third-party libraries is leading to inevitable results.

Obviously, the next leap will be sophisticated supply chain attacks based upon poisoning AI.

e.g. The difficulty curve for writing apps went down a bit when iOS became popular, but nowadays iOS 17 is probably more complex than Snow leopard was in 2009 so there aren’t any low hanging fruit left for the median SWE.

I have to disagree with this.

There seems to be this weird mindset in tech that because there's problem X (the Five Eyes countries hacking each others' citizens at each others' request, Meta collecting data on users, a massive attack using xz that almost got into the wild, etc.) that China isn't a problem. It's this strange "our house isn't in order because of our own doing, so it doesn't matter if some dude off the street starts squatting in it" idea.

If you have a country that has a tech legacy mainly related to espionage and attacks on other countries' systems - and make no mistake, that's China's main legacy - don't buy their stuff, no matter how many times it's said that it's fine. At some point it won't be fine.

You can fix that and better secure FOSS projects; it's not one or the other.

I guarantee you more services link to glibc, and that glibc has a much, much larger attack surface than anything systemd.

Many people have warned that systemd does way too much for an init system, and presents way too much attack surface. Cluck, cluck (chickens coming home to roost).

Firstly, it's not true: most systemd services don't link to libsystemd. It's useful for supporting the sd_notify feature, but plenty of things support it without using libsystemd.

Rich attack surface: there was like... four dependencies that aren't glibc, and irrespective of this backdoor attempt they were almost all already being made dlopen dependencies, which would've prevented this backdoor from working since it would've never been triggered for sshd. (Note, also, that linking libsystemd into sshd was a downstream patch.)

Seriously:

    $ libtree $(nix build --print-out-paths nixpkgs\#systemdLibs)/lib/libsystemd.so
    libsystemd.so.0 
     └── libcap.so.2 [runpath]

The truth is that libsystemd and systemd in general don't have ridiculous dependency trees or attack surfaces. Most likely the big reason why this backdoor was being pushed heavily to try to make it into Debian and Fedora releases was because the clock was ticking on the very small in they had found in the first place.

There are a lot of criticisms of systemd that I think are reasonable, but this really isn't one.

I feel like a lot of people are really glossing over this point. What's happened here is that Red Hat and Debian have made a choice to patch OpenSSH in a way that opened it up to this attack.

It's a little ironic that e.g. Arch, which actually shipped the malicious code to end users since it publishes changes so much faster, as shipped never would have executed the payload (because they didn't patch OpenSSH).

The backdoor was designed to only be injected during the building of an RPM or Debian package. Arch never would have been impacted no matter what choices they made. They were trying to hit production systems while minimizing their potential exposure to other less important types of users beforehand.

The most intensive operation I can think of that might happen at sshd startup would be host key generation, but at least in NixOS it looks like this is actually handled in an ExecStartPre= script, which I believe means it will happen before After= units execute.

Of course I sincerely doubt that distributions decided to add sd_notify support just for the hell of it, so I am sure there is a reason, it's just not overtly obvious, especially considering plenty of distros don't do this and sshd (and dependent units presumably) certainly seems to work absolutely fine.

Maybe libsystemd shouldn't provide sd_notify then, if you're not supposed to use it?

Generally, I think using libsystemd is probably a good idea, if you are a C program that wishes to have support for sd_notify. Even better, even before they moved things to dlopen, the dependency chain was very clean; nobody would've batted an eye at the idea that something, even something as critical as the SSH daemon, would have a very popular and well-regarded compression library which is already trusted in many package managers loaded into the process space. There is absolutely nothing unreasonable about considering that to be trustworthy. Your sshd probably has zlib linked in and it is fine.

Seriously. We're not talking about the difference between zero dependency and one dependency, but rather something more like 34 dependencies vs 38 dependencies. Here's my sshd, noting that libtree excludes some glibc stuff by default:

    $ libtree `which sshd`
    /run/current-system/sw/bin/sshd 
    ├── libgssapi_krb5.so.2 [runpath]
    │   ├── libkrb5.so.3 [runpath]
    │   │   ├── libk5crypto.so.3 [runpath]
    │   │   │   ├── libkrb5support.so.0 [runpath]
    │   │   │   │   ├── libkeyutils.so.1 [runpath]
    │   │   │   │   └── libresolv.so.2 [runpath]
    │   │   │   ├── libkeyutils.so.1 [runpath]
    │   │   │   └── libresolv.so.2 [runpath]
    │   │   ├── libcom_err.so.3 [runpath]
    │   │   │   ├── libkrb5support.so.0 [runpath]
    │   │   │   ├── libkeyutils.so.1 [runpath]
    │   │   │   └── libresolv.so.2 [runpath]
    │   │   ├── libkrb5support.so.0 [runpath]
    │   │   ├── libkeyutils.so.1 [runpath]
    │   │   └── libresolv.so.2 [runpath]
    │   ├── libk5crypto.so.3 [runpath]
    │   ├── libcom_err.so.3 [runpath]
    │   ├── libkrb5support.so.0 [runpath]
    │   ├── libkeyutils.so.1 [runpath]
    │   └── libresolv.so.2 [runpath]
    ├── libkrb5.so.3 [runpath]
    ├── libcom_err.so.3 [runpath]
    ├── libk5crypto.so.3 [runpath]
    ├── libz.so.1 [runpath]
    ├── libcrypto.so.3 [runpath]
    │   └── libpthread.so.0 [runpath]
    ├── libldns.so.3 [runpath]
    │   ├── libssl.so.3 [runpath]
    │   │   ├── libcrypto.so.3 [runpath]
    │   │   └── libpthread.so.0 [runpath]
    │   └── libcrypto.so.3 [runpath]
    └── libpam.so.0 [runpath]
        └── libaudit.so.1 [runpath]

Of course, I'm sure Debian and Fedora have their reasons for the specific patch that adds sd_notify support, it probably does improve something somewhere, but this incident absolutely showcases how the consequences of innocuous-looking patches like that can domino into something absolutely devastating in a non-trivial manner. I strongly suspect whatever it improves on would not have been worth it if the backdoor had succeeded in proliferating. Nothing is a panacea, but you definitely gain some advantage by sticking to configurations supported by the upstream where possible.

That said: Debian and Fedora already (largely) dodged this bullet, libsystemd already (even before this news) plugged the hole that made it possible in the first place, and everything is fine for now. Debian and Fedora thus have no real reason to remove this patch other than if their sensibilities regarding it have changed, because the risk that it posed is basically gone now. Now for this specific patch to pose a threat, you'd have to compromise either libsystemd or libcap2. On any distro that uses systemd to begin with, compromising systemd already gets you root, and libcap2 IIRC is maintained by the Linux kernel folks, so of course if anyone compromises that, it's way, way game over already.

Is this some grand lesson about systemd bloat potentially enabling a horrible sshd backdoor? You can read it that way, but it's a pretty silly take, considering all of the factors. But if it makes people who get emotionally attached to hating init systems happy, more power to them I suppose. As for normal risk analysis, there is no particular factor here that completely dominates for what really enabled this. It was a combination of decisions that, on their own, are completely justifiable and reasonable, but when combined, led to near disaster. Not a new story and not one that's going to stop occurring either...

10000s lines of boiler plate code? Very easy to skip something in there. Why do we need to repeat everything?

The problem is exactly that.. preprocessing, build scripts etc.

Everything is a script and is therefore executable instead of a configuration file/statement which only increases the attack surface

But what init system you use is irrelevant to the issue of what do you do when an upstream project is taken over by a malicious developer?

I don't think most people mind system as a service manager, if that's all it did.

> port, socket,

That's inetd.

> timed triggers

That's cron.

> etc?

Etc.

What could possibly go wrong. I'm sure there's no history of compression tools having serious vulnerabilities due to implementation errors...

[1] https://seclists.org/oss-sec/2024/q1/268

Disclaimer: I did not search for all possible occurrences of dlopen().

> libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.

Other than that I did try a manual port (of zstd) to Java but I was not pleased with the results.

The other part is that systemd uses plain unix sockets with the most basic of protocols (that part along with docker forwarder was doable)

[By dismantling the car completely and reassembling it, since you'd have to rebuild from source...]

I get the idea of systemd and I do think that the Unix principles were good for the 70s but maybe not so great today

Still in the true LP fashion, a lot of design decisions are not being thought with stability or longevity in mind

With systemd, its a big problem because systemd is widely distributed. Systemd failed on its promise to create a new standard for defining services. Right now a ton of projects ship their own supervisor (runit, supervisord) or docker compose file to reduce their contact surface to systemd. Look what GitLab Omnibus does.

But with everything connected to systemd (udev, dbus), anything without systemd is sort of second class in terms of being tested. Ideally i would have the stability of Debian without the surprises from systemd. I tell people to "Press Ctrl+Alt+Del 7 times within two seconds" way too often.

https://en.m.wikipedia.org/wiki/Lennart_Poettering

Your hastiness to hate on systemd made you forget what we're even talking about…

But why in tarnation they need libxz to talk to systemd over dbus? Harm reduction should start there

And also maybe let's rethink dbus or bring it to a modern standard and not be just Corba for geeks.

We should be grateful it got caught so quickly. I sent Andres a honest thank you email. It isn't financial (and I am just one individual) but it felt the least I could do.

If there is a way to donate, I would. This person could've earned more via HackerOne or black market and instead went with an arguably better path. I don't think we can compete with the latter though, unless we start treating this for what it is: a weapon.

Like, what would be different if the software would be closed source and developers payed by companies? I think it would be at least as hard to notice such an exploit and sometimes it might be easier (if the company is located in your jurisdiction).

Maybe the current mindset of assembling programs could be improved. There is a trend in some architecture to separate everything in their own container and while I don't think it can be directly applied everywhere that model gives more separation for cases like this. Engineering is an art of trade-offs and maybe now we can afford making different trade-offs than 30 years ago (when some things where decided)

DJB’s qmail, written in 1995, was made of 5 distinct processes, owned by different users, with no trust among them. Coincidentally, it was the MTA with the best security record for more than a decade (and also the most efficient one).

It would have likely had a similar record even if it was only a monolithic process - because DJB - but it was built as 5 processes so even if one falls, the others do not.

It's endemic not just to open source. macOS has supported app sandboxing since Snow Leopard (2009), yet virtually no application outside the App Store (where sandboxing is mandatory) sandboxes itself. App sandboxing could stop both backdoors from supply chain compromises and vulnerabilities in applications in their tracks. Yet, developers put their users at risk and we as users cheer when a developer moves their application out of the App Store.

It's time for, not only better funding, but significantly better security than the 70ies/80ies Unix/Windows models.

Usually applications distributed outside the app store would simply not work (or be limited to close to useless) if sandboxed.

My pet example these days, DaisyDisk, cannot show what takes 10-30% of my space in the app store version. And can't delete protected files in Applications etc.

Which would be nice if it were a malicious free to play game, but it's an application that graphically reports what's taking space on your computer and optionally deletes stuff that you've chosen. So it simply can't work well inside the sandbox.

I disagree. Sure, there are some applications that need to be distributed outside the App Store because they need additional privileges (like DaisyDisk), but there are many applications that are distributed outside the app store that could be sandboxed. Just to give some examples, why do Discord, Signal, Obsidian, Dash, or 1Password have unsandboxed processes? (1Password was in the App Store and sandboxed before it became an Electron app.)

Discord asks for the accessibility option to read system wide keystrokes for push to talk. Can sandboxed apps do that?

Also, no matter how secure the app store is, i'd very much like to be able to install applications without -ing Apple's permission. So having something available in the app store doesn't give me a warm fuzzy feeling.

You can register a shortcut in the sandbox, you cannot read system-wide keystrokes from the sandbox.

The main issue is that the tooling around sandboxing is poor. If there are violations the best you're going to get is an opaque message logged to an unbelievably verbose log torrent. Also, the low level sandboxing APIs you need to really batten things down for internal components aren't documented. Chrome uses them anyway, and they just have to rely on the fact that they're so big that Apple won't break them. But if you're smaller, it's a risk.

Just two (similar) examples that cross my mind: large monopolies (ex: thinking Microsoft in the 90's), imbalanced power between actors (ex: only nations states/huge companies can do something not a small group of hackers).

I think diversification and accessibility of the technology would solve more problems overall rather than just focusing on security. It is just hard to strike a balance between efficiency and diversity (ex: one Linux distribution might be efficient resource-wise but it is not diverse; how many and how different would be diverse enough?)

Thing is, in most companies, is cheaper and more efficient to deal with a sporadic vulnerability than having your e-mail system DOSed every other week.

This is the kind of compromises that normal people and companies have to do all the time, but radicals and cryptopunks like DJB can't seem to understand. Sure, he's a brilliant mathematician and cryptographer, but his grasp of reality outside academia seems very flimsy, IMO.

At least with open source we are able to detect them.

Also, let's not forget that this particular backdoor was initially found through behavioral analysis, not by reading the source code. I think Linus' law "given enough eyeballs, all bugs are shallow" has been refuted. Discovering bugs does not scale linearly with eyeballs, you need the right eyeballs. And the right eyeballs are often costly.

If your implicit premise that having the source code available makes it easier to analyze code than closed source, you can also flip the argument around: it is easier for bad actors to find exploitable vulnerabilities because the source code is available.

(Note: I am a staunch supported of FLOSS, but for different reasons, such as empowerment and freedom.)

Example: https://en.wikipedia.org/wiki/NSAKEY

I think currently even Android, iOS, and ChromeOS have far better security than most desktop and server OSes. I think of the widely-used general purpose OSes, only macOS comes fairly close because it adopted a lot of things from iOS (fully verified boot, sealed system partition, app sandboxing for App Store apps, protected document/download/mail/... directories, etc.).

However the incentives even if a company detected the infiltration is to keep quiet about it. Lets say that a closed source "accessd" was backdoored, a database admin notices the new version (accessd 2024.6 SAAS version model 2.0+ with GPT!) is slower than the previous version, they put it down to enshittificaiton. Or they contact the company who has no incentive to spend money to look into it. There's no way the database admin (or whoever) can look into the git commit chain and see where the problem happened.

[0] https://rigor-mortis.nmrc.org/@simplenomad/11218486968142017...

Unfortunately, the sarcastic tone starts to become a barrier to separating signal from noise about halfway through. Okay, you’re super clever, the NSA is a threat, too, we get it. Security vendors are largely hucksters, fine. What were we talking about again?

"the issues around tens of thousands of unpaid developers writing the core of the world’s most critical infrastructure nowadays."

We need to take opensource software sustainability much more seriously.

For everyone else? Not much different to open source.

What if the entire company is a bad actor?

Although, I can imagine a sub-sub-contractor identity being somewhat easily forged, it's still harder than just creating a GitHub account.

Very doubtful in my eyes. Very few companies have the strict validation which would be required to catch this.

Good validation that goes into central components is often skimped on arbitrary deadlines, which companies are full of. On something tangential like this I suspect nobody would really notice.

I can also hear the "I noticed now there's an extra delay which isn't supposed to be there, can I investigate?" "Sorry, but this is not critical for this deadline" agile mentality.

On my current jig developer churn is not high, yet I've only recently met developers hired 6+ months ago. I know first-hand only a handful of the committers I see. Barely know the most common commiters. I generally do watch commits of the trees/projects I'm interested into, but I'm a minority, and such behavior wouldn't catch something similar to the xz situation unless I'm absolutely lucky.

This also ignores the fact that you can just as well corrupt a current employee.

And corrupting an employee doesn't sound that easy, either. I mean, we do get paid above average.

That still leaves shit third party contractors and compromising employees computers/accounts, though.

Some underpaid indian contractor you mean?

I'm sure they'd never ever possibly accept a bribe!

They push wildly complicated and performance-costly bandaids for vanishingly unlikely exploits for dubious benefit, and pay very little attention to holes like this you can drive a truck through. https://xkcd.com/538/ is very fitting.

The fact a database hacker found it, and it missed all these supposed scanners and auditors and security experts is the icing on the cake. But even if it was one of them that found it, it's already in the wild.

The technical programming tools and techniques for security are important don't get me wrong, the security industry just seems to have very little concept of cost-benefit, the big picture of how attackers and their targets operate and what motivates them. They seem to mostly exist to justify their own existence.

I've already seen "security experts" wheel out the big scary (and meaningless) term "nation state" over this, which makes me laugh. Sure it could have been the CCP. It could also have just been some bored kid in his parents basement, that just wouldn't look good to admit though.

The public availability of the software helped catch the attack much faster than commercial software. Even if there was intensive scrutiny of changes to the project, a person familiar with the process can still come up with hard to detect backdoors.

Future backdoors may be stealthier but what this case demonstrated is that even a database hacker who doesn't do security audits could catch it simply because it's opensource. The expectation should be, such backdoors would be detected many months or years after the fact if they were your typical closed source popular application.

This is a case for open source software usage and funding. The security industry can't do much in terms of prevention against a malicious insider that knows the codebase more than any outsider. And opensource or not, people can get paid or implanted to sabotage software.

It's not always obvious but devs adding backdoors and vulns is not all that new.

The guy may have been anonymous here but a legit dev's github account compromise could lead to the same outcome.

Each open source project decides how much vetting is applied to contributors. I don't think you can contribute to Linux without using your real name and email for example. In some countries, getting a job for the express purpose of sabotage is very common. People using stolen id's to get remote dev job's is also a thing (although I haven't heard that being abused for backdooring). At least with open source, you can audit the code for anonymous user contributions and look at their policy for it.

It's been reported that both Fedora and Ubuntu's valgrind rigs flagged this, actually, which is why the package changes hadn't propagated to those distros yet. It's true that they didn't get as far as recognizing the root cause because Andres beat them to it, but the security infrastructure was absolutely doing its job.

It wasn't just that they were beat fair and square on equal terms -- they had opportunity before the packages got into their distro-upstream, and they (allegedly) are the ones who look for and audit security issues, the database guy found it by observing some peripheral performance issue it caused, and pursued and tracked down the problem. A stark contrast to the uncurious attitude of the security theater that was supposed to be actively looking for these things and examining warnings from tools.

Again, that happened only in Debian testing and Fedora rawhide (and maybe a few other downstreams, though really the exploit requires a particular systemd setup and won't work on arbitrary "linux" systems). Those are rolling release variants deliberately intended to take upstreams rapidly with minimal review, precisely so that integration testing can be done. And it was, and it flagged the issue via at least one symptom.

Only one person gets the cookie for finding any given problem, usually. And this time it was Andres, and we should absolutely celebrate that. But that doesn't mean we run off and shit on the losers, right?

Right, lots of failures happened. I'm not sure the point.

> Those are rolling release variants deliberately intended to take upstreams rapidly with minimal review, precisely so that integration testing can be done. And it was, and it flagged the issue via at least one symptom.

And nothing was done about it.

> Only one person gets the cookie for finding any given problem, usually. And this time it was Andres, and we should absolutely celebrate that. But that doesn't mean we run off and shit on the losers, right?

The problem isn't that they were not the first to find it, the problem is that they weren't even in the race. And people and processes are not be immune to criticism for failures, so we absolutely can "shit on" the failures at many levels that have helped to make this situation possible.

could have become bored, forgot and then "rediscovered" it

play the long game, low and slow.

A nation state would utilize this attack chain to steal copious mounts of sensitive data and prepare infrastructure for coordinated attacks on critical infrastructure and intellectual property.

The threat is relevant because it informs us about their strategy. If you expert security experts to prevent this, they need to know why they’re preventing and have some concept of the realism of any given notional threat. There’s aren’t enough of us to address every threat.

And I really disagree that it matters, whether it's a bored kid, an underhanded corporation, an organized crime ring, a terrorist group, or a government security agency, doesn't really matter that much. Some understanding of motivation sure helps, but they don't need to play CIA agent. It's not like the PLA would use this type of hole, the FSB a different type, ISIS would go some completely different route, etc. They're all just looking for ways they can infiltrate, sabotage, deny, steal, etc. So just work on the security problems, roughly with priority from the cheapest cost to benefit, to the greatest.

You don't need to know anything about nation states or any of that claptrap to know that rogue maintainers and contributors are a major problem. And you don't even have to know that to know that linking your privileged sshd process to libsystemd is a stupid idea. Yet millions are spent on ever more esoteric and complicated hardening and speculation issues and things that have never been shown to stop feasible remote attacks.

Probably because those are cool and techy and working on people problems is much harder and less interesting to many tech types.

Not to say don't do any of the technical stuff, but the calls for more funding of OSS I'm hearing won't solve problems like this if the funding goes to more of that kind of thing. It's not a funding problem it's a spending problem. What is desperately needed is some exceptional people who have big picture understanding of the problem, including good people-skills, to hold the purse strings and set some direction.

But he wasn't actively looking for security bugs nor gives off security researcher hubris.

My worry is, how many have not been spotted and have not been averted? From my reading of this one, it was blind luck that it was spotted.

Clicked over Andre’s profile and saw he works at Microsoft. I am a bit surprised that no article or HN post about this prevented hack mentions this fun fact.

(Yes we should still buy this guy a coffee but he’s also a principle engineer at MS)

or how much damage they`ve done in the past to said landscape due to poor security.

Not every OpenSSH build it's linked against xz. OpenBSD's one isn't.

However, to my knowledge nobody has fully analyzed the malware binary yet. So we don't know whether it only contains a single mechanism to attack sshd or whether it also has other harmful components.

And correct, this was not an issue for Slackware Current (which has xz 5.6.1). Slackware 15 has xz 5.2.5

It feels like we should have better socialized defense against this kind of thing. When I see a name pop up online, I want an immediate readout on how active this person is over time and where. We need better tools to get high level views of who we are dealing with.

>> odd valgrind complaint in automated testing of postgres

This implies these kinds of complaints are routinely ignored.

This isn't a good take. This presents proprietary software as offering superior assurances against backdoors, if only it were practical to completely avoid Free and Open Source software. The truth is approximately the opposite: code being Free and Open Source should be considered a necessary condition, but not a sufficient one, for resisting backdoors.

> We should start funding every open source project! Yeah, good luck with that.. I’ll start saving for my trip to Mars.

This kind of defeatism is unhelpful.

In the image below that remark the author makes clear that proprietary software is often subverted.

Ultimately, the attacker deemed systemd to be common enough to consider it as the way to hijack sshd, but indirect enough to avoid discovery by audit. It says something about software monoculture.

However they left the door open to future payloads by simply dropping new blobs into the test cases folder.

Considering it's a library that isn't even a direct dependency of ssh and someone was willing to put in over two years of effort, it doesn't seem crazy to start wondering what other (small) projects have been suffering maintainer fatigue or are even just big enough where a "lgtm" response is almost guaranteed by a sane looking PR.

"Smuggle in unexpected code/blobs as test files", "bastardise make scripts to inject your code into the build" and then "have your code preferentially override [decryption] routines" aren't just (as of last week) unexpected, but would also seemingly be usable against any other project out there.

I'd suggest it's more like finding and killing a cockroach in your bathroom. Thinking that the roach problem is now gone for good and was only confined to the one room seems... optimistic at best.

I also think it's fun to wildly speculate on what else could be affected - or to play the game another way, "if I had control over sshd, what else would I want?"

From an outside point of view, I'd quite like web servers. They're often externally accessible when ssh isn't, are doing cert validation/compression/decryption and I'd wager on most boxes, could easily give you an ssl wrapped tunnel to (the vulnerable) sshd running on the servers loopback address without selinux blocking you.

Actually, having written that, has anyone looked at the SSH _client_ process rather than server? Surely it needs to be able to parse certs in the same way as the server does, and getting clients to leak their private keys somewhere seems like it would be extremely useful as well.