This is outstanding... I really wish I would have had this learning systemd as it would have saved me _hours_. Let's be real - a lot of Linux fundamental stuff is still pretty terse. Learning tools like this really smooth it out and optimize how long it takes to wrap your head around a concept. Also, I am a _huge_ advocate for hands-on in regards to retention.
Adding to my favorites and will be passing this on over the years - thank you for such a great resource.
If anything the biggest problem the systemd docs have is verbosity. It's all classic UNIX man pages, a billion pages of detail on every possible setting with no useful examples anywhere. Fortunately the core system is simple enough that the learning curve isn't too steep but I'd really hate to try and learn it from the official docs. They don't even apply CSS to the HTML versions of the docs.
This isn't a knock but it's exactly what I was talking about and why I find OP's learning tool to be so valuable.
Lots of folks learn by example and hands-on labs. Personally, I'd much rather learn the basic ropes by jumping into a tool like OP's vs. finding/digging through all of these resources. I'll also criticize to say you likely already know much about systemd, and were able to pull/filter these resources much easier vs someone completely new to the concepts.
To illustrate further: vim is another tool that has outstanding learning resources, everything from very quick "hey get started" examples docs all the way up to adventure games. If I had to go back and relearn vim I would absolutely do it this way vs. digging on man pages like when I was a kid in the 90's. Personally, I learn by doing.
---
Overall - OP's thingy is what I would call a "rich interactive learning tool." It's anecdotal, and obvious projection - but _for me_, interactive learning tools optimize the time it takes to fully "grok" a subject from scratch vs. jumping into a bunch of docs/man pages.
TIL about that manpage. I've always been guessing at which level in the hierarchy would contain a particular option ("Is it service-specific? Or maybe applies to all execs? Or maybe applies to all units?") and just giving up and opening them all in parallel.
Couldn't agree more. Once familiar with the big picture this page is the best starting point whenever to you want change something.
To get the big picture I read the systemd posts on https://0pointer.net/blog/archives.html But that's 10 years already! No idea what would be recommended today.
And almost none of the tools help you naturally discover the missing feature you didn't know to solve the problem in the most "systemd*" way.
The docs are not friendly enough to introduce you to new tools, and they're mostly shallow enough to not explain the technical details of how things interact or where all the defaults are stored. They exist but clearly this page exists because they need to improve.
And, as a Debian user who often finds himself on the Arch docs - there's a ton of distro-specific stuff in the Arch wiki. And rightfully so - it's a wiki for a distro.
These sorta landmines when trying to just research/digest a concept can really suck. OP's tool really eloquently breaks things down to _just_ core concepts so you can quickly start to grok what I consider to be a relatively complex tool.
In my experience very little of it is distro-specific, especially if you're on a distro that has similar components as Arch. I'm on OpenSUSE Tumbleweed, which also uses systemd, etc and is a rolling distro with the latest versions of everything, so a lot of the wiki articles apply directly as long as I translate the package names.
Agree - but there's still bits and pieces that don't translate over to other distros which can suck if you're jumping into to learning something. When I read the Arch Wiki as a Debian user I realize I'm reading it through the lens of someone using a different distro.
Overall I use the Arch wiki very often and it's because of the exact point you're making - I'm just being pedantic saying those slight distro differences can be a pain.
There is a trillion of tutorials for every young and hyped technology out there: Svelte/FastAPI/Elm etc. However try to learn how to maintain and secure a server or a database or anything fundamental... and it can be really hard to find good and updated material.
Agreed, I have generally avoided systemd as I don't feel it's fully baked yet (not trying to start a flame war, that is just my experience and I'm sure it's fine for others). Still, I need to really learn it both for when I am ready to use it daily and just to have it in my wheelhouse for supporting others. A guide like this seems like a great way to go "hands on" without spinning up an instance and fumbling through it blindly.
Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
>given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
I don't understand where anyone is getting this impression from, that describes the opposite of my experience with systemd. The authors publish stability guarantees, and they have done so for several years: https://systemd.io/PORTABILITY_AND_STABILITY/
I love systemd both for it's actually functionality and the fact that it's almost like the flagship of a whole movement to make Linux more consistent and integrated and less of a pile o scripts.
But for some reason, the authors occasionally say something that feels completely out of place for the project. They fix important stuff eventually, but sometimes it gets confusing to watch them discuss it.
The "Invalid username runs a service as root" bug is a good example of something that they didn't seem to take seriously. But it's fixed now.
Also, PulseAudio. Pulse is good. It's not that great and was deployed before it's ready on some distros. The whole Pulse/Jack split that is only just now being fixed was annoying.
Some people might still judge Poettering based on 10yo bugs in Pulse rollout?
After a lot of recent events, I don't feel comfortable judging any open source project on the speed at which they fix bugs or the way they communicate about it. In open source there is no obligation for anyone to fix anything or communicate anything. To me it's a miracle when anything gets fixed at all. It feels like the community is still largely held together with duct tape and string.
Yeah, I just expect all software to be buggy. To be honest my main way of evaluating software is by Googling stuff like "X ate my data" or "X stopped working" and seeing how many major complaints people have, relatively to how many users there are.
I think the duct tape and string feeling comes from the obsession with smallness and modularity and people trying to make "Linux not be like windows".
Half the FOSS scene WANTS things to be taped together because their main thing is random tinkering.
A guarantee is only as good as the willingness to honor it. There are great ideas and some good implementations in systemd, but when the people behind it have a pattern of refusing to fix (and sometimes even acknowledge) show-stopping bugs, it makes me apprehensive about relying on it.
Again, I didn't want my comment to start a heated discussion about systemd, but inevitably that happens. If I had praised it someone else would have come along to tell me how much it sucks (I don't think it sucks, personally, I just have issues with its current development process).
-----
Adding "hacks" but not testing to ensure said hacks don't cause issues:
Lennart Poettering 2014-02-21 13:49:25 UTC
To make this work we'd need a patch, as nobody of us tests this.
Comment 3 Michael Shigorin 2014-04-04 06:30:57 UTC
Hope all of you either test all the combinations or do not break at will those you don't have the time and inclination to test, at least in system-wide components that are not specific to systemd, while pushing the latter hard.
Comment 4 Lennart Poettering 2014-04-04 14:56:43 UTC
Well, cgroups-less kernels are explicitly not supported by systemd. However we added some hacks to allow it to boot to a certain degree even if a lot of things will not work correctly afterwards. In this mode when you boot you will actually get a warning on screen and bootup is delayed by 10s to make sure the user understands this.
Now, this mode recently broke, and it will segfault early on. I am happy to take a patch to 'fix' this again, but I will not work on this as i dont run kernels like this, and as mentioned its not really supported anyway...
Another option is to simply be honest amd stop supporting in entirely, and refuse booting completely. And I figure this is what I will eventually do if nobody cares enough to send me a patch to fix that segfault.
I can't figure out what the problem here is, are you trying to run systemd with a kernel with cgroups compiled out? I don't see how that's a show stopping bug, from the discussion there it was never supported.
No, I'm not. I'm satisfying the other person's request for sources with examples I've come across in the past. In this case it was not about cgroup-less kernels but about Poettering's and Sievers' attitudes towards fixing the issue.
This is why I don't like to talk about systemd on any internet forum; it always leads to people like you ready to attack any position around the subject, and me having to defend a position I don't even hold (I don't think systemd sucks, I think its developers could do better though). I am done with this thread.
I'm not attacking you or asking you to defend a position, I'm asking for clarification because I legitimately do not understand what the issue is. I totally agree with you, it is not helping anyone to become defensive over this. I don't understand what the attitude was, it seems to have been fixed 8 years ago? Did I miss something?
>Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.
The core dependency principles around targets, services and startup does not see a lot of change, so this concern is not really realistic. You'll see more of this around the supporting utilities (networkd, resolved, and so on).
In fairness, quite often I've found this is often a case of exposing existing brokenness rather than creating new problems, but it doesn't make it any more comfortable when it happens.
That doesn't follow. If anything, I would expect knowing, for practical reasons, you have to learn something you think is dumb would make you even more angry about it, not less... or, maybe, you just believe something being either popular or common means it must be good? :(
It's reasonable to have the opinion that juggling chainsaws is a dumb idea, even before you've learnt the techniques for making it slightly less dangerous.
Is that really any dumber than doing any similarly dangerous industrial jobs that require a lot of training? Or something else dangerous in entertainment like being a Hollywood stuntman, or being Siegfried & Roy? I'm sure you could get a lot of money for having that skill if you did it in the right place.
It's on version 250. How many stable releases do you need to feel that it is "fully baked"? The software is of course evolving and new features are added but breaking changes are rare.
Adding to my favorites and will be passing this on over the years - thank you for such a great resource.