Wandering Thoughts archives

(Not) changing the stop timeout for systemd login session scopes

I wrote earlier about an irritating systemd reboot behavior, where systemd may twiddle its thumbs for a minute and a half before killing some still-running user processes and actually rebooting your machine. In that entry I suggested that changing DefaultTimeoutStopSec in /etc/systemd/user.conf (or in a file in user.conf.d) would fix this, and then reversed myself in an update. That still leaves us with the question: how do you change this for user scopes?

The answer is that you basically can't. As far as I can tell, there is no way in systemd to change TimeoutStopSec for just user scopes (or just some user scopes). You can set DefaultTimeoutStopSec in /etc/systemd/system.conf (or in a file in system.conf.d), but then it affects everything instead of just user scopes.

(It's possible that you're okay with this; you may be of the view that when you say 'shut the machine down', you want the machine down within a relatively short period even if some daemons are being balky. And aggressively killing such balky processes is certainly traditional Unix shutdown behavior; systemd is being really quite laid back here. This is an appealing argument, but I haven't had the courage to put it into practice on my machines.)

As you can see in systemctl status's little CGroup layout chart, your user processes are in the following inheritance tree:

user.slice -> user-NNN.slice -> session-N.scope

You can create systemd override files for user.slice or even user-NNN.slice, but they can't contain either DefaultTimeoutStopSec or TimeoutStopSec directives. The former must go in a [Manager] section and the latter must go in a [Service] one, and neither section is accepted in slice units. There is a 'user@NNN.service' that is set up as part of logging in, and since it's a .service unit you can set a TimeoutStopSec in systemd override files for it (either for all users or for a specific UID), but it isn't used for very much and what you set for it doesn't affect your session scope, which is where we need to get the timeout value set.

If you want to use a really blunt hammer you can set KillUserProcesses in /etc/systemd/logind.conf (or in a logind.conf.d file). However, this has one definite and one probable drawback. The definite drawback is that this kneecaps screen, tmux, and any other way of keeping processes running when you're not logged in, unless you always remember to use the workarounds (and you realize that you need them in each particular circumstance).

(I don't know what it does to things like web server CGIs run as you via suexec or processes started under your UID as part of delivering mail. Probably all of those processes count as part of the relevant service.)

The probable drawback is that I suspect systemd does this process killing in the same way it does it for reboots, which means that the default 90 second timeout applies. So if you're logging in with processes that will linger, you log out, and then you immediately try to reboot the machine, you're still going to be waiting almost the entire timeout interval.

An irritating systemd behavior when you tell it to reboot the system

For reasons well beyond the scope of this entry, I don't use a graphical login program like gdm; I log in on the text console and start X by hand through xinit (which is sometimes annoying). When I want to log out, I cause the X server to exit and then log out of the text console as normal. Now, I don't know how gdm et al handle session cleanup, but for me this always leaves some processes lingering around that just haven't gotten the message to give up.

(Common offenders are kio_http_cache_cleaner and speech-dispatcher and its many friends. Speech-dispatcher is so irritating here that I actually chmod 700 the binary on my office and home machines.)

Usually the reason I'm logging out of my regular session is to reboot my machine, and this is where systemd gets irritating. Up through at least the Fedora 24 version of systemd, when it starts to reboot a machine and discovers lingering user processes still running, it will wait for them to exit. And wait. And wait more, for at least a minute and a half based on what I've seen printed. Only after a long timer expires will systemd send them various signals, ending in SIGKILL, and force them to exit.

(Based on reading manpages it seems that systemd sends user processes no signals at all at the start of a system shutdown. Instead it probably waits TimeoutStopSec, sends a SIGTERM, then waits TimeoutStopSec again before sending a SIGKILL. If you have a program that ignores everything short of SIGKILL, you're going to be waiting two timeout intervals here.)

At one level, this is not crazy behavior. Services like database engines may take some time to shut down cleanly, and you do want them to shut down cleanly if possible, so having a relatively generous timeout is okay (and the timeout can be customized). In fact, having a service have to be force-killed is (or should be) an exceptional thing and means that something has gone badly wrong. Services are supposed to have orderly shutdown procedures.

But all of that is for system services and doesn't hold for user session processes. For a start, user sessions generally don't have a 'stop' operation that gets run explicitly; the implicit operation is the SIGHUP that all the processes should have received as the user logged out. Next, user sessions are anarchic. They can contain anything, not just carefully set up daemons that are explicitly designed to shut themselves down on demand. In fact, lingering user processes are quite likely to be badly behaved. They're also generally considered clearly less important than system services, so there's no good reason to give them much grace period.

In theory systemd's behavior is perhaps justifiable. In practice, its generosity with user sessions simply serves to delay system reboots or shutdowns for irritatingly long amounts of time. This isn't a new issue with systemd (the Internet is full of complaints about it), but it's one that the systemd authors have let persist for years.

(I suspect the systemd authors probably feel that the existing ways to change this behavior away from the default are sufficient. My view is that defaults matter and should not be surprising.)

When I started writing this entry I expected it to just be a grump, but in fact it looks like you can probably fix this behavior. The default timeout for all user units can be set in /etc/systemd/user.conf with the DefaultTimeoutStopSec setting; set this down to less than 90 seconds and you'll get a much faster timeout. However I'm not sure if systemd will try to terminate a user scope other than during system shutdown, so it's possible that this setting will have other side effects. I'm tempted to try it anyways, just because it's so irritating when I slip up and forget to carefully kill all of my lingering session processes before running reboot.

Update: I'm wrong. Setting things in user.conf does nothing for the settings you get when you log in.

(You can also set KillUserProcesses in /etc/system/logind.conf, but that definitely will have side effects you probably don't want, even if some people are trying to deal with them anyways.)

I should learn more about Grub2

I have a long-standing dislike of Grub2 (eg, and). Ever since I started having to deal with it I've felt that it's really overcomplicated, and this complexity makes it harder to deal with. There's a lot more to know and learn with Grub2 than there is with the original Grub, and I resent the added complexity for what I feel should be a relatively simple process.

You know what? The world doesn't care what I think. Grub2 is out there and it's what (almost) everyone uses, whether or not I like it. And recent events have shown me that I don't know enough about how it works to really troubleshoot problems with it. As a professional sysadmin, it behooves me to fix this sort of a gap in my knowledge for the same reason that I should fix my lack of knowledge about dpkg and apt.

I'm probably never going to learn enough to become an expert at Grub 2 (among other things, I don't think there's anything we do that requires that much expertise). Right now what I think I should learn is twofold. First, the basic operating principles, things like where Grub 2 stores various bits of itself, how it finds things, and how it boots. Second, a general broad view of the features and syntax it uses for grub.cfg files, to the point where I can read through one and understand generally how it works and what it's doing.

(I did a little bit of this at one point, but much of that knowledge has worn off.)

Unfortunately there's a third level I should also learn about. Grub2 configurations are so complex that they're actually mostly written and updated by scripts like grub2-mkconfig. This means that if I want to really control the contents of my grub.cfg on most systems, I need to understand broadly what those scripts do and what they get controlled by (and thus where they may go wrong). Since I don't think this area is well documented, I expect it to be annoying and thus probably the last bit that I tackle.

(If I cared about building custom grub2 configurations, it should be the first area. But I don't really; I care a lot more about understanding what Grub2 is doing when it boots our machines.)

Ubuntu 16.04's irritatingly broken MySQL updates

I tweeted:

So, Ubuntu 16.04 can't apply MySQL server updates if you have the server installed but have disabled it running. Good show, you lot.

We install the MySQL server package on a few machines but deliberately don't start the daemon. In older versions of Ubuntu, this worked reasonably well; you could do it, you could keep the daemon from starting on boot, and you could apply updates (although doing so generally started the daemon up, so you had to remember to then go stop it). In 16.04, if you've disabled the daemon your attempts to apply updates will error out:

mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11

The direct cause of this problem is that the mysql-server-5.7 postinstall script needs to run mysql_upgrade, which requires the server to be running. Perhaps at this point you sigh, run 'service mysql start', and try the upgrade again. It'll still fail, because the postinstall script is more complicate and more wrong than that.

The postinstall script needs to stop the MySQL daemon, do some things, and then start the daemon again and run mysql_upgrade (and then restart the daemon yet again). It does all of this starting and restarting by running invoke-rc.d, and invoke-rc.d specifically refuses to start disabled daemons. In the grand Unix tradition, this behavior is burried in an innocuous phrasing in the invoke-rc.d manpage:

invoke-rc.d is a generic interface to execute System V style init script /etc/init.d/name actions, obeying runlevel constraints as well as any local policies set by the system administrator.

Via a complex chain of actions, what 'obeying runlevel constraints' translates to here is that if you do 'systemctl disable <whatever>', invoke-rc.d will decided that <whatever> is specifically blocked from running and not start it.

(Invoke-rc.d in general is the wrong tool in Ubuntu 16.04, because it's actually fairly tied to the System V init framework. The system goes through ugly hacks in order to make it work 'right' on things that are actually native systemd .service units, as the MySQL daemon is.)

This selectivity is the wrong approach, or at least it's in the wrong place. What the postinst script should really be doing is unconditionally shutting down the server, unconditionally starting it to run mysql_upgrade, unconditionally shutting it down again, and only then using invoke-rc.d to conditionally start it again. This would achieve the twin goals of upgrading MySQL while not leaving the daemon running if it's disabled. This would actually be an improvement over the 14.04 situation, instead of a massive headache.

(Of course I expect that the real answer is simply that no one thought about this possibility, and that if we were to file a bug report we'd be told that disabling the daemon is not a supported configuration.)

The workaround is simple. Before you try to apply a MySQL server pack update, do 'systemctl enable mysql'. After it's done, do 'systemctl disable mysql; systemctl stop mysql' to return to the original state.

Sidebar: 'Runlevel constraints' and invoke-rc.d

Invoke-rc.d checks to see whether something is enabled or disabled by looking for S* and K* symlinks in /etc/rc<runlevel>.d. In 16.04, the 'runlevel' is arbitrary and is reported as '5', so we're looking at /etc/rc5.d. When you do 'systemctl disable' or 'systemctl enable' on an Ubuntu 16.04 system and the service also has an /etc/init.d file, systemctl helpfully maintains rcN.d S* and K* symlinks for you. So running 'systemctl disable mysql' also creates a /etc/rc5.d/K02mysql, which invoke-rc.d will then see as saying that mysql is specifically constrained to not start in runlevel 5, and so should not be started.

(If there was no /etc/rc5.d symlink at all, invoke-rc.d would also conclude that it shouldn't start the MySQL daemon.)

An interesting (and alarming) Grub2 error and its cause

I upgraded my office workstation from Fedora 23 to Fedora 24 today, following my usual procedure of doing a live upgrade with dnf. Everything went smoothly, which is normal, and it was pretty fast, which isn't my normal experience but was probably because my root filesystem is now on SSDs. After the updates finished, I ran the grub2-install command that you're instructed to do and rebooted. My machine made it into Grub's menu but trying to start any kernel immediately halted with an error about the symbol grub_efi_secure_boot not being found (as in this Fedora 24 bug report or this old Ubuntu one).

This could politely be called somewhat alarming. Since it seemed to involve (U)EFI booting in some way, I went through the BIOS settings for my motherboard to see if I could turn that off and force a pure BIOS boot to make things work. Naturally I wound up looking through the boot options screen, at which point I noticed that the boot order looked a little odd. The BIOS's boot list didn't have enough room to display full names for drives, but the first and second drives had names that started with 'ST31000', and things called 'Samsung ...' were way down the list at the bottom.

At this point the penny dropped: my BIOS was still booting from my older hard drives, from before I'd moved the root filesystem to the SSDs. The SSDs were definitely considered sda and sdb by Linux and they're on the first two SATA links, but the BIOS didn't care; as far as booting went, it was sticking to its old disk ordering. When I'd updated the Grub2 boot blocks with grub2-install, I'd of course updated the SSD boot blocks because that's what I thought I was booting from; I hadn't touched the HD boot blocks. As a result the old Fedora 23 Grub boot blocks were trying to load Fedora 24 Grub modules, which apparently doesn't work very well and is a classic cause of these Grub 'undefined symbol' errors.

Once I realized this the fix was pleasantly simple; all I had to do was put the SSDs in their rightful place at the top of the (disk) boot priority list. Looking at the dates, this is the first Fedora version upgrade I've done since I added the SSDs, which explains why I didn't see it before now.

There's an argument that the BIOS's behavior here is sensible. If I'm correct about what's going on, it has essentially adopted a 'persistent boot order' in the same way that Linux (and other Unixes) are increasingly adopting persistent device names. I can certainly see people being very surprised if they add an extra SSD and suddenly their system fails to boot or boots oddly because the SSD is on a channel that the BIOS enumerates first. However, it's at least surprising for someone like me; I'm used to BIOSes cheerfully renumbering everything just because you stuck something into a previously unused SATA channel. A BIOS that doesn't do that for boot ordering is a bit novel.

(This may be especially likely on motherboards with a mix of 6G and 3G SATA ports. You probably want the 6G SATA ports enumerated first, and even if HDs live there for now, they're going to wind up being used for SSDs sooner or later.)

In the process of writing this entry I've also discovered that while I moved my root filesystem over to the SSDs, I seem to never have moved /boot; it's still a mirrored partition on the HDs. I'm not sure if this was something I deliberately planned, if I was going to move /boot later but forgot, or if I just plain overlooked the issue. I have some notes from my transition planning, but they're silent on this.

(Since /boot is still on the HDs, I'm now uncertain both about how the BIOS is actually numbering my drives and how Grub2 is finding /boot. Maybe the Grub boot blocks (technically the core image) have a hard-coded UUID for /boot instead of looking at specific BIOS disks.)

A feature I wish the Linux NFS client had: remapping UIDs and GIDs

My office workstation is a completely standalone machine, deliberately not dependent on, say, our NFS fileserver infrastructure. As a sysadmin's primary machine, there are obvious motivations for this and it's not something I'm ever going to change, but at the same time it does have its drawbacks and it'd be nice to sometimes NFS mount my fileserver home directory on my workstation.

Unfortunately there is a little but important obstacle. For peculiar historical reasons, the UID and GID of my local workstation account are not the same as on our fileservers (and thus our Ubuntu servers and so on). My login name is the same so things like SSH work fine, but NFS cares about the actual UID and GID and I wind up out of luck as far as NFS mounts go.

In theory the solution to this is NFS v4. In practice we don't use NFS v4 now, we're very unlikely to add NFS v4 any time soon for general use, and there is exactly zero chance that we'll add NFS v4 to our environment just so that I can NFS mount my fileserver home directory on my office workstation. Put bluntly, there are much easier solutions to that particular problem, ones that put all the work on my head where it rightfully belongs.

Hence my wish that the NFS client would support remapping UIDs and GIDs between the NFS server's view and the local Unix's view. In my particular situation I'd even be happy with a mount option that said 'always tell the server that we're UID X and GID Y', because that's all I need.

There's a pro and a con argument for doing this in the NFS client instead of the NFS server. The pro argument is that it's easier to scale this administratively if you can do it in the client. If it's done on the client, only the person running the client has to care; if it's done on the server, the server administrators have to be involved every time another client needs another UID or GID remapping.

The con argument is that NFS v3 'security' is already leaky enough without handing people a totally convenient client side way of subverting it totally (well, if you have root on a client). Yes, sure, you can already do this in a number of ways if you have client root, but all of those ways take at least some work. This feature would make it trivially easy, and there's a benefit to avoiding that.

(I expect that the real answer here is 'the Linux NFS maintainers have no interest in adding NFS v3 UID and GID mapping code in either the client or the server; use NFS v4 if you need this'. On the other hand, they did add NFS v3 server support for more than 16 GIDs.)