The bad side of systemd: two recent systemd failures

December 12, 2014

In the past I've written a number of favorable entries about systemd. In the interests of balance, among other things, I now feel that I should rake it over the coals for today's bad experiences that I ran into in the course of trying to do a yum upgrade of one system from Fedora 20 to Fedora 21, which did not go well.

The first and worst failure is that I've consistently had systemd's master process (ie, PID 1, the true init) segfault during the upgrade process on this particular machine. I can say it's a consistent thing because this is a virtual machine and I snapshotted the disk image before starting the upgrade; I've rolled it back and retried the upgrade with variations several times and it's always segfaulted. This issue is apparently Fedora bug #1167044 (and I know of at least one other person it's happened to). Needless to say this has put somewhat of a cramp in my plans to upgrade my office and home machines to Fedora 21.

(Note that this is a real segfault and not an assertion failure. In fact this looks like a fairly bad code bug somewhere, with some form of memory scrambling involved.)

The slightly good news is that PID 1 segfaulting does not reboot the machine on the spot. I'm not sure if PID 1 is completely stopped afterwards or if it's just badly damaged, but the bad news is that a remarkably large number of things stop working after this happens. Everything trying to talk to systemd fails and usually times out after a long wait, for example attempts to do 'systemctl daemon-reload' from postinstall scripts. Attempts to log in or to su to root from an existing login either fail or hang. A plain reboot will try to talk to systemd and thus fails, although you can force a reboot in various ways (including 'reboot -f').

The merely bad experience is that as a result of this I had occasion to use journalctl (I normally don't). More specifically, I had occasion to use 'journalctl -l', because of course if you're going to make a bug report you want to give full messages. Unfortunately, 'journalctl -l does not actually show you the full message. Not if you just run it by itself. Oh, the full message is available, all right, but journalctl specifically and deliberately invokes the pager in a mode where you have to scroll sideways to see long lines. Under no circumstance is all of a long line visible on screen at once so that you may, for example, copy it into a bug report.

This is not a useful decision. In fact it is a screamingly frustrating decision, one that is about the complete reverse of what I think most people would expect -l to do. In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this.

(Oh, and journalctl goes out of its way to set up this behavior. Not by passing command line arguments to less, because that would be too obvious (you might spot it in a ps listing, for example); instead it mangles $LESS to effectively add the '-S' option, among other things.)

While I'm here, let me mention that journalctl's default behavior of 'show all messages since the beginning of time in forward chronological order' is about the most useless default I can imagine. Doing it is robot logic, not human logic. Unfortunately the systemd journal is unlikely to change its course in any significant way so I expect we'll get to live with this for years.

(I suppose what I need to do next is find out wherever abrt et al puts core dumps from root processes so that I can run gdb on my systemd core to poke around. Oh wait, I think it's in the systemd journal now. This is my unhappy face, especially since I am having to deal with a crash in systemd itself.)

Comments on this page:

By Dave at 2014-12-12 10:46:52:

Thank you for your post. I too dislike `journalctl`.

That being said... FYI, you can alter the behavior of `less` at runtime. In this case, simple type `-S` (that's [dash] [shift-S]) while viewing a file. This will toggle line-wrapping.

By Brian at 2014-12-12 12:32:15:

Also, I just checked the source code and you can override systemd's silly pager defaults (and silly overriding of $LESS if present) with $SYSTEMD_LESS. So:


gets you a line-wrapped, colored pager.

By dozzie at 2014-12-12 17:26:25:

You could always pipe the less output through cat. In the case that STDOUT is redirected, less gives up and just passes the text through.

By gsauthof (gsauthof) at 2014-12-13 09:45:42:

I agree, journalctl has terrible defaults.

At least, in current journalctl versions, one can override the less settings via SYSTEMD_LESS (e.g. SYSTEMD_LESS=FRXMK)- which is documented in journalctl(1) - e.g. on Fedora 21.

Unfortunately, the CentOS 7 journalctl version seems to be too old, i.e. SYSTEMD_LESS is not documented there and journalctl does not pick the setting.

By Chris at 2014-12-13 17:30:55:

#env SYSTEMD_LESS=FRXMK journalctl -lb

Written on 12 December 2014.
« What good kernel messages should be about and be like
There are two parts to making your code work with Python 3 »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Dec 12 01:51:53 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.