> libsystemd is linked to all systemd servies, which opens a rich attack surface of third party services to backdoor.
Many people have warned that systemd does way too much for an init system, and presents way too much attack surface. Cluck, cluck (chickens coming home to roost).
Firstly, it's not true: most systemd services don't link to libsystemd. It's useful for supporting the sd_notify feature, but plenty of things support it without using libsystemd.
Rich attack surface: there was like... four dependencies that aren't glibc, and irrespective of this backdoor attempt they were almost all already being made dlopen dependencies, which would've prevented this backdoor from working since it would've never been triggered for sshd. (Note, also, that linking libsystemd into sshd was a downstream patch.)
The truth is that libsystemd and systemd in general don't have ridiculous dependency trees or attack surfaces. Most likely the big reason why this backdoor was being pushed heavily to try to make it into Debian and Fedora releases was because the clock was ticking on the very small in they had found in the first place.
There are a lot of criticisms of systemd that I think are reasonable, but this really isn't one.
> (Note, also, that linking libsystemd into sshd was a downstream patch.)
I feel like a lot of people are really glossing over this point. What's happened here is that Red Hat and Debian have made a choice to patch OpenSSH in a way that opened it up to this attack.
It's a little ironic that e.g. Arch, which actually shipped the malicious code to end users since it publishes changes so much faster, as shipped never would have executed the payload (because they didn't patch OpenSSH).
>It's a little ironic that e.g. Arch, which actually shipped the malicious code to end users since it publishes changes so much faster, as shipped never would have executed the payload (because they didn't patch OpenSSH).
The backdoor was designed to only be injected during the building of an RPM or Debian package. Arch never would have been impacted no matter what choices they made. They were trying to hit production systems while minimizing their potential exposure to other less important types of users beforehand.
Honestly, I'm a bit curious why sshd startup would ever be significant. Not sure.
The most intensive operation I can think of that might happen at sshd startup would be host key generation, but at least in NixOS it looks like this is actually handled in an ExecStartPre= script, which I believe means it will happen before After= units execute.
Of course I sincerely doubt that distributions decided to add sd_notify support just for the hell of it, so I am sure there is a reason, it's just not overtly obvious, especially considering plenty of distros don't do this and sshd (and dependent units presumably) certainly seems to work absolutely fine.
So on the one hand libsystemd provides what both you and the manual page describe as "useful" functionality for implementing this feature, but on the other you're implying that Debian shouldn't have used it to do what it's designed for?
Maybe libsystemd shouldn't provide sd_notify then, if you're not supposed to use it?
No, I never said you should or shouldn't use sd_notify. I said that Debian and Fedora added it downstream, that it was not a decision of OpenSSH.
Generally, I think using libsystemd is probably a good idea, if you are a C program that wishes to have support for sd_notify. Even better, even before they moved things to dlopen, the dependency chain was very clean; nobody would've batted an eye at the idea that something, even something as critical as the SSH daemon, would have a very popular and well-regarded compression library which is already trusted in many package managers loaded into the process space. There is absolutely nothing unreasonable about considering that to be trustworthy. Your sshd probably has zlib linked in and it is fine.
Seriously. We're not talking about the difference between zero dependency and one dependency, but rather something more like 34 dependencies vs 38 dependencies. Here's my sshd, noting that libtree excludes some glibc stuff by default:
But there is one thing that is questionable, and that's whether or not downstreams like Debian and Fedora should really be making patches like this that add new dependencies to security-critical programs like OpenSSH. It's one thing if OpenSSH takes on this dependency itself, but downstreams adding it is scarier specifically because it's so unexpected. If you didn't know the way OpenSSH was packaged specifically in Debian and Fedora, you would have absolutely no way of figuring out that liblzma5 could be in its process space and thus it wouldn't be part of the threat model. Certainly the upstream developers would have zero chance of ever noticing this.
Of course, I'm sure Debian and Fedora have their reasons for the specific patch that adds sd_notify support, it probably does improve something somewhere, but this incident absolutely showcases how the consequences of innocuous-looking patches like that can domino into something absolutely devastating in a non-trivial manner. I strongly suspect whatever it improves on would not have been worth it if the backdoor had succeeded in proliferating. Nothing is a panacea, but you definitely gain some advantage by sticking to configurations supported by the upstream where possible.
That said: Debian and Fedora already (largely) dodged this bullet, libsystemd already (even before this news) plugged the hole that made it possible in the first place, and everything is fine for now. Debian and Fedora thus have no real reason to remove this patch other than if their sensibilities regarding it have changed, because the risk that it posed is basically gone now. Now for this specific patch to pose a threat, you'd have to compromise either libsystemd or libcap2. On any distro that uses systemd to begin with, compromising systemd already gets you root, and libcap2 IIRC is maintained by the Linux kernel folks, so of course if anyone compromises that, it's way, way game over already.
Is this some grand lesson about systemd bloat potentially enabling a horrible sshd backdoor? You can read it that way, but it's a pretty silly take, considering all of the factors. But if it makes people who get emotionally attached to hating init systems happy, more power to them I suppose. As for normal risk analysis, there is no particular factor here that completely dominates for what really enabled this. It was a combination of decisions that, on their own, are completely justifiable and reasonable, but when combined, led to near disaster. Not a new story and not one that's going to stop occurring either...
While I agree generally about systemd, this line in particular is not even correct. You don’t have to link libsystemd to run as a systemd service. Using the notify feature is optional anyway, but you can do it without linking to that library.
Would an OpenBSD-style minimum functionality Unix philosophy designed init system stop a developer from taking over as maintainer of a different upstream project, allowing them to submit malicious patches?
The grandparent comment was about how the issue was actually systemd. My comment is pushing back against that. As you point out, that isn’t a simple swap out. The replacement to systemd can be as complicated and intertwined as systemd is today if not more.
But what init system you use is irrelevant to the issue of what do you do when an upstream project is taken over by a malicious developer?
> The grandparent comment was about how the issue was actually systemd.
It's not my view that "the issue was actually systemd". The issue was a complex, sophisticated attack by someone taking a long view. But the attacker injected his code into the system by manipulating part of systemd into loading a fairly obscure compression library that he had hacked. Systemd is of course more-or-less omnipresent these days.
Why does an init system need to load libraries at all? Well, it doesn't, unless it has arrogated to itself much more functionality than PID1 should ever have.
The dependency is attributable, in the largest part, to systemd's neoplastic aggrandizement of userland infrastructure and associated plumbing, making this a distinction without much of a difference.
This is another furphy, because OpenSSH proper neither requires nor uses xz/lzma. It's made clear in Andres Freund's original report¹ that the libsystemd dependency dragging it along arises from distros patching openssh to support systemd notifications. The sad part is that systemd notifications are just a datagram on a socket, so using libsystemd for this is reminiscent of Joe Armstrong's banana.
I've seen that ambit claim too, but I'm not even sure what distro(s) it is referring to since I'm unable to confirm it on any host where I have ldd casually to hand. Ref however https://seclists.org/oss-sec/2024/q1/356
That packaging error makes liblzma being pulled in at installation (well, it's probably already there if pid 1 requires it). But it will not make the sshd binary use it. So I think the original claim stands: Without patching sshd for the notification it will not use liblzma.
Disclaimer: I did not search for all possible occurrences of dlopen().
> libselinux does not link to liblzma. It turns out the confusion was because of an old downstream-only patch in Fedora and a stale dependency in the RPM spec which persisted long-beyond its removal.
Most cars no longer come with a spare wheel, certainly by default. Not sure if that’s because cars don’t get punctures any more due to the great state of the roads here in the U.K, or because people just can’t even change a wheel.
Its acceptable for avahi and pulseaudio because you don't need to use them.
With systemd, its a big problem because systemd is widely distributed. Systemd failed on its promise to create a new standard for defining services. Right now a ton of projects ship their own supervisor (runit, supervisord) or docker compose file to reduce their contact surface to systemd. Look what GitLab Omnibus does.
But with everything connected to systemd (udev, dbus), anything without systemd is sort of second class in terms of being tested. Ideally i would have the stability of Debian without the surprises from systemd. I tell people to "Press Ctrl+Alt+Del 7 times within two seconds" way too often.
Many people have warned that systemd does way too much for an init system, and presents way too much attack surface. Cluck, cluck (chickens coming home to roost).