The bad side of systemd: two recent systemd failures
In the past I've written a number of favorable entries about systemd.
In the interests of balance, among other things, I now feel that I
should rake it over the coals for today's bad experiences that I
ran into in the course of trying to do a
yum upgrade of one system
from Fedora 20 to Fedora 21, which did not go well.
The first and worst failure is that I've consistently had systemd's master process (ie, PID 1, the true init) segfault during the upgrade process on this particular machine. I can say it's a consistent thing because this is a virtual machine and I snapshotted the disk image before starting the upgrade; I've rolled it back and retried the upgrade with variations several times and it's always segfaulted. This issue is apparently Fedora bug #1167044 (and I know of at least one other person it's happened to). Needless to say this has put somewhat of a cramp in my plans to upgrade my office and home machines to Fedora 21.
(Note that this is a real segfault and not an assertion failure. In fact this looks like a fairly bad code bug somewhere, with some form of memory scrambling involved.)
The slightly good news is that PID 1 segfaulting does not reboot
the machine on the spot. I'm not sure if PID 1 is completely stopped
afterwards or if it's just badly damaged, but the bad news is that
a remarkably large number of things stop working after this happens.
Everything trying to talk to systemd fails and usually times out
after a long wait, for example attempts to do '
from postinstall scripts. Attempts to log in or to
su to root from
an existing login either fail or hang. A plain
reboot will try to
talk to systemd and thus fails, although you can force a reboot in
various ways (including '
The merely bad experience is that as a result of this I had occasion
journalctl (I normally don't). More specifically, I had
occasion to use '
journalctl -l', because of course if you're going
to make a bug report you want to give full messages. Unfortunately,
journalctl -l does not actually show you the full message.
Not if you just run it by itself. Oh, the full message is available,
all right, but
journalctl specifically and deliberately invokes
the pager in a mode where you have to scroll sideways to see long
lines. Under no circumstance is all of a long line visible on screen
at once so that you may, for example, copy it into a bug report.
This is not a useful decision. In fact it is a screamingly frustrating
decision, one that is about the complete reverse of what I think most
people would expect
-l to do. In the grand systemd tradition, there is
no option to control this; all you can do is force
journalctl to not
use a pager or work out how to change things inside the pager to not do
journalctl goes out of its way to set up this behavior. Not
by passing command line arguments to
less, because that would be too
obvious (you might spot it in a
ps listing, for example); instead
$LESS to effectively add the '
-S' option, among other
While I'm here, let me mention that
journalctl's default behavior
of 'show all messages since the beginning of time in forward
chronological order' is about the most useless default I can imagine.
Doing it is robot logic, not human logic.
Unfortunately the systemd journal is unlikely to change its course
in any significant way so I expect we'll get to live
with this for years.
(I suppose what I need to do next is find out wherever
al puts core dumps from root processes so that I can run
systemd core to poke around. Oh wait, I think it's in the
systemd journal now.
This is my unhappy face, especially since I am having to deal with
a crash in systemd itself.)