Putting cron jobs into systemd user slices

September 2, 2017

In my last installment on fair share scheduling with systemd and Ubuntu 16.04, I succeeded in working out how to get ordinary user processes (ones spawned from people logging in or sshing in or the like) organized into the right cgroup hierarchy so they would be subjected to per-user fair share scheduling. However, I discovered and noted a limitation that is relevant for our environment, which is that in a standard Ubuntu 16.04 system, processes started by cron are not put into user slices; instead they all run under the cron.service system slice. A commentator suggested that this could probably be fixed with the PAM systemd module, and I got sufficiently interested in this to work out how to do it.

The important bit of PAM magic is the pam_systemd PAM module. The manpage writeup implicitly focuses on actual login sessions of some form (including ssh command execution), but in fact it works for everything and does what you'd expect. If pam_systemd is one of the session modules, whatever 'session' is created through that PAM service will put processes into a session scope inside a user-${UID} slice that is itself under user.slice. If general per-user fair share scheduling is enabled, this will cause these processes to be part of the user's fair-share scheduling.

(As the pam_systemd manpage implies in passing, this may also have some side effects depending on logind.conf settings. This may constrain your ability to use this for, say, cron jobs in some environments.)

One of the things that happens in our environment is that we run a lot of root cron jobs for things that need to run frequently like our password propagation system. Unfortunately pam_systemd seems to cause a small burst of logging every time it's used, at least on Ubuntu 16.04, so having root cron jobs spawn new session scopes every time they run may be a pain (and you may not want some of the side effects for root jobs, like having them be per-user fair-share scheduled). Helpfully PAM provides us a way around this via the pam_succeed_if module. So we can put the following in /etc/pam.d/cron to only force use of systemd session scopes and user slices for cron jobs run by actual users:

session [default=1 success=ignore] pam_succeed_if.so quiet uid > 999
session optional     pam_systemd.so

(The normal starting user UID on Ubuntu 16.04 is UID 1000. Your local first user UID may be different, and I confess that ours certainly is.)

A daring person could put this in /etc/pam.d/common-session-noninteractive instead, which on a standard Ubuntu 16.04 machine is included by the PAM files atd, cron, samba, sudo, and systemd-user (which is used when you run 'systemd --user', not that you normally do). Having looked at this list, I think I would only put it in cron and atd.

(Yes, we have some users who (still) use at.)

All of this implicitly exposes a fundamental limitation of systemd per-user fair share scheduling, but that's going to have to be another entry.

Comments on this page:

From at 2017-09-02 08:15:18:

Yes, limiting pam_systemd to individual configs (at & cron) is a good idea.

The systemd-user config already uses (and requires) it, so you'd have a duplicate; Samba doesn't need it much (although you could perhaps use it to limit I/O); and giving it to su/sudo can have unexpected results. (What would happen if a service tried to use su with this module enabled? It would accidentally end up creating a logind session, and escaping the service's original cgroup.)

From at 2017-09-02 08:25:08:

Ah, and regarding

(which is used when you run 'systemd --user', not that you normally do)

It's true for RHEL/CentOS (which, I believe, has completely removed this feature out of their systemd builds), but in many other distributions you'll see a systemd --user instance being started automatically upon login, for every user. This, I think, is what you've already noticed on Ubuntu 16.

That is actually the main reason why I mentioned that distros are reluctant to use pam_systemd for at/cron: if the user wasn't already logged in interactively, then every cron job would cause user@<uid>.service to be started and stopped – possibly with all the associated per-user units. That's not just a burst of logging, but possibly a burst of unexpected CPU/IO/resource usage as well. (Even people who do know and use ~/.config/systemd/user/ probably won't expect that to be invoked on every cron job.)

(It could be that pam_systemd has an opt-out; perhaps class=background – I have not tested that one yet...)

Written on 02 September 2017.
« With git, it's useful to pick the right approach to your problem
A fundamental limitation of systemd's per-user fair share scheduling »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Sep 2 00:42:08 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.