2017-09-02
Putting cron jobs into systemd user slices
In my last installment on fair share scheduling with systemd and
Ubuntu 16.04, I succeeded in working
out how to get ordinary user processes (ones spawned from people
logging in or sshing in or the like) organized into the right cgroup
hierarchy so they would be subjected to per-user fair share scheduling.
However, I discovered and noted a limitation that is relevant for
our environment, which is that in a standard Ubuntu 16.04 system,
processes started by cron are not put into user slices; instead
they all run under the cron.service system slice. A commentator
suggested that this could probably be fixed with the PAM systemd module, and I got
sufficiently interested in this to work out how to do it.
The important bit of PAM magic is the pam_systemd PAM
module. The manpage writeup implicitly focuses on actual login sessions
of some form (including ssh command execution), but in fact it works
for everything and does what you'd expect. If pam_systemd is one of
the session modules, whatever 'session' is created through that PAM
service will put processes into a session scope inside a user-${UID}
slice that is itself under user.slice. If general per-user fair
share scheduling is enabled, this will
cause these processes to be part of the user's fair-share scheduling.
(As the pam_systemd manpage implies in passing, this may also have some side effects depending on logind.conf settings. This may constrain your ability to use this for, say, cron jobs in some environments.)
One of the things that happens in our environment is that we run a
lot of root cron jobs for things that need to run frequently like
our password propagation system.
Unfortunately pam_systemd seems to cause a small burst of logging
every time it's used, at least on Ubuntu 16.04, so having root cron
jobs spawn new session scopes every time they run may be a pain
(and you may not want some of the side effects for root jobs, like
having them be per-user fair-share scheduled). Helpfully PAM provides
us a way around this via the pam_succeed_if module.
So we can put the following in /etc/pam.d/cron to only force use
of systemd session scopes and user slices for cron jobs run by
actual users:
session [default=1 success=ignore] pam_succeed_if.so quiet uid > 999 session optional pam_systemd.so
(The normal starting user UID on Ubuntu 16.04 is UID 1000. Your local first user UID may be different, and I confess that ours certainly is.)
A daring person could put this in
/etc/pam.d/common-session-noninteractive instead, which on a
standard Ubuntu 16.04 machine is included by the PAM files atd,
cron, samba, sudo, and systemd-user (which is used when you
run 'systemd --user', not that you normally do). Having looked
at this list, I think I would only put it in cron and atd.
(Yes, we have some users who (still) use at.)
All of this implicitly exposes a fundamental limitation of systemd per-user fair share scheduling, but that's going to have to be another entry.