== The bad side of systemd: two recent systemd failures

In the past I've written a number of favorable entries about systemd.
In the interests of balance, among other things, I now feel that I
should rake it over the coals for today's bad experiences that I
ran into in the course of trying to do a _yum_ upgrade of one system
from Fedora 20 to Fedora 21, which [[did not go well
https://twitter.com/thatcks/status/543167790166052864]].

The first and worst failure is that I've consistently had systemd's
master process (ie, PID 1, the true init) segfault during the upgrade
process on this particular machine. I can say it's a consistent
thing because this is a virtual machine and I snapshotted the disk
image before starting the upgrade; I've rolled it back and retried
the upgrade with variations several times and it's always segfaulted.
This issue is apparently [[Fedora bug #1167044
https://bugzilla.redhat.com/show_bug.cgi?id=1167044]] (and I know
of at least one other person it's happened to). Needless to say
this has put [[somewhat of a cramp
https://twitter.com/thatcks/status/543200559990595585]] in my plans
to upgrade my office and home machines to Fedora 21.

(Note that this is a real segfault and not an assertion failure.
In fact this looks like a fairly bad code bug somewhere, with some
form of memory scrambling involved.)

The slightly good news is that PID 1 segfaulting does not reboot
the machine on the spot. I'm not sure if PID 1 is completely stopped
afterwards or if it's just badly damaged, but the bad news is that
a remarkably large number of things stop working after this happens.
Everything trying to talk to systemd fails and usually times out
after a long wait, for example attempts to do '_systemctl daemon-reload_'
from postinstall scripts. Attempts to log in or to _su_ to root from
an existing login either fail or hang. A plain _reboot_ will try to
talk to systemd and thus fails, although you can force a reboot in
various ways (including '_reboot -f_').

The merely bad experience is that as a result of this I had occasion
to use _journalctl_ (I normally don't). More specifically, I had
occasion to use '_journalctl -l_', because of course if you're going
to make a bug report you want to give full messages. Unfortunately,
~~'_journalctl -l_ does not actually show you the full message~~.
Not if you just run it by itself. Oh, the full message is *available*,
all right, but _journalctl_ specifically and deliberately invokes
the pager in a mode where you have to scroll sideways to see long
lines. Under no circumstance is all of a long line visible on screen
at once so that you may, for example, copy it into a bug report.

This is not a useful decision. In fact it is a screamingly frustrating
decision, one that is about the complete reverse of what I think most
people would expect _-l_ to do. In the grand systemd tradition, there is
no option to control this; all you can do is force _journalctl_ to not
use a pager or work out how to change things inside the pager to not do
this.

(Oh, and _journalctl_ goes out of its way to set up this behavior.  Not
by passing command line arguments to _less_, because that would be too
obvious (you might spot it in a _ps_ listing, for example); instead
it mangles _$LESS_ to effectively add the '_-S_' option, among other
things.)

While I'm here, let me mention that _journalctl_'s default behavior
of 'show all messages since the beginning of time in forward
chronological order' is about the most useless default I can imagine.
Doing it is [[robot logic ../unix/UnixAnnoyance]], not human logic.
Unfortunately [[the systemd journal is unlikely to change its course
in any significant way SystemdDislike]] so I expect we'll get to live
with this for years.

(I suppose what I need to do next is find out wherever _abrt_ et
al puts core dumps from root processes so that I can run _gdb_ on
my _systemd_ core to poke around. Oh wait, [[I think it's in the
systemd journal now
http://www.freedesktop.org/software/systemd/man/coredumpctl.html]].
This is my unhappy face, especially since I am having to deal with
a crash in systemd itself.)