Systemd's slowly but steadily increased idealism about the world

February 6, 2017

When it started out, systemd was in many ways relentlessly pragmatic. My shining example of this is that the developers went to fairly great lengths to integrate both System V init scripts and /etc/fstab into systemd in a fairly deep and thus quite convenient way. The easy way would have been to just run things and mount filesystems through some compatibility shims and programs. Systemd went the extra distance to make them more or less real units, which means that you can do things like add extra dependencies to System V init scripts through /etc/systemd/system overrides, just as if they were native systemd units.

(This has not always worked seamlessly, especially for mounts, but it has gotten better over time.)

As well as being convenient for people using systemd, I suspect that this was a pragmatic decision. Being a better System V init than SysV init itself undoubtedly didn't hurt systemd's case to be the winning init system; it gave people a few more reasons to like systemd and approve of it and maybe even push for it.

Unfortunately, since then systemd developers have shown an unfortunate and increasing streak of idealism. More and more, systemd seems not to be interested in dealing with the world as it actually is, with all of its grubby inconvenient mess; instead it simply assumes some pure and ideal version of things. If the world does not measure up to how it is supposed to be, well, that is not systemd's problem. Systemd will do the nominally right thing no matter how that works out in practice, or doesn't.

Exhibit one for this is how systemd interprets LSB dependencies in System V init scripts. These dependencies are predictably wrong in any number of cases, because they've never been really used before. Ignoring them and just running init scripts in order (with some pauses in the middle) would be the pragmatic choice, but instead systemd chose the idealistic one of 'we will assume that declared dependencies are correct, gain a minor speed boost, and if things blow up it's not our fault'.

Exhibit two for me is the general non-resolution of our SATA port multiplier issue with device naming. The general systemd view seems to be that this is not their problem; either it should be in some vague diffuse other system that no one is writing today, or the kernel's sysfs should provide more direct information, or both. In no case is this going to be solved by systemd. Never mind that systemd is getting things blatantly wrong; it is not their problem to fix, even though they could. This once again is clear idealism and purity triumphing over actual usability on actual hardware and systems.

It seems clear to me that systemd is less and less a pragmatic system where the developers are willing to make compromises and go out of their way to deal with the grubby, messy world as it actually is, and more and more a project where the developers want to deal with a pure world where things are all done in the philosophically right way. We all know how this ends up, because we have seen this play out in security, among other places. If you're not solving problems in the real world, you're not really solving problems; you are just being smug and 'clever'.

(This elaborates on and explains an old tweet of mine.)

PS: Or perhaps systemd was always sort of like this, and I didn't really notice it before. You do need more than a little bit of idealism to think 'we will do an init system right this time', and certainly systemd had some idealistic beliefs right from the start. Socket activation and (not) handling things that wanted to depend on the network being up are the obvious cases. Systemd was arguably correct but certainly kind of annoying about them.


Comments on this page:

By dozzie at 2017-02-07 05:21:45:

[...] systemd chose the idealistic one of 'we will assume that declared dependencies are correct, gain a minor speed boost, and it things blow up it's not our fault'.

This is actually stupid. Expecting that a maintainer will enumerate in advance all the dependencies and the list will be correct is ridiculous and has never worked reliably; always something slipped somewhere.

systemd with its socket activation promised that the dependencies will be detected a) automatically and b) from what makes them dependencies in the first place (one service uses another), what makes the whole idea sound, but from what I see it doesn't even try to live up to these expectations.

By Roger Leigh at 2017-02-07 15:00:38:

To comment on your assertion that the LSB dependencies were untested. This is absolutely not the case, certainly for Debian, Ubuntu and SuSE.

Debian removed numbered runlevels years ago, and fully switched over to dependency-based boot by default at least three major releases back. insserv was used to scan the headers, build a dependency graph and than create an ordering for booting in the correct order (including accounting for parallelism). startpar could even boot the system up starting services in parallel. This was tested over many years.

I was one of the sysvinit maintainers and worked on this stuff for several years, and I was the person who excised the last traces of numeric ordering. There may well have been random leaf packages with bad dependencies, but all the core scripts and packages were well reviewed and tested.

Roger

You wrote:

Or perhaps systemd was always sort of like this, and I didn't really notice it before.

IMHO it was always sort of like this. For example, see this WONTFIX bug about journald from 2014:

Lennart Poettering: Yupp, journal corruptions result in rotation, and when reading we try to make the best of it. they are nothing we really need to fix hence.

So losing one's logs is something that can't be helped. Of course if journald's file format was ACID, it wouldn't be an issue, but they'll have none of that. The above caused Howard Chu of OpenLDAP to write the following:

I've been watching the systemd drama with a skeptical eye, from a distance. But tonight I read something that crystallized it all for me, so now I'm compelled to point out why Lennart Poettering and systemd cannot be trusted.

Just to reiterate that the LSB stuff worked very well in Debian. Roger Leigh's work (and others) was very high quality. Come back Roger, we miss you!

This goes way further back. I remember PulseAudio problems on my ancient ThinkPad, which was current technology at that time. When you asked the hardware for a 44100Hz sample rate it didn't give you back exactly 44100Hz, probably due to some detail of the audio chip's clocks or something. If you pushed data constantly to /dev/dsp, it worked fine, just tuned very slightly sharp, so all other players worked fine. However, PulseAudio ignored the different reported rate and the conventional flow-control of /dev/dsp, and assumed 44100Hz and pushed data at that rate regardless, causing constant underflow. So the sound output was a distorted mess. (This was bad practice even with hardware supporting 44100Hz, because it assumes that the CPU timer and sound card timer are perfectly synchronized, which is impossible, so it would have caused underflow glitches for everyone, just less noticeably). The bug was reported (not by me), and Pottering's response (IIRC) was basically: Go and get some proper hardware, won't fix, not a bug.

From that experience alone (never mind all that has happened since) I am still holding out against systemd. I came to Linux for the positive cooperative environment, not for this.

David Magda writes,

For example, see this WONTFIX bug about journald from 2014...

Yes, and see the explanation immediately after that. The TLDR is that what systemd is doing now is the very best behaviour: move the file out of the way (but still keep it in the set of files read by journalctl when you're printing logs), never change the corrupted file again , and use code in journald on reading to reconstruct any corruption as best as possible.

This avoids bugs in a "fixer" program making the corruption worse, and also means that when you read the file later you're using the newest, best fixer rather than whatever existed at the time the file was written.

To me this clearly seems the right way to go.

Written on 06 February 2017.
« Our advantage in reliable backups is that we get restore requests
How to see and flush the Linux kernel NFS server's authentication cache »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Mon Feb 6 22:46:15 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.