How we implement per-user CPU and memory resource limits on Ubuntu
A while back I wrote about imposing temporary CPU and memory limits on a user, using cgroups and systemd's features to fiddle around with them. Since then we have wound up with a number of shared general use machines where we've decided it's wiser to impose limits on everyone all of the time, so that one person can't blow up a general use server through either excessive CPU usage or excessive memory usage. We've done this on Ubuntu 16.04 and now 18.04, and with some limitations it works well to keep our systems from having too many problems.
We've found that at least on 18.04, it's impossible to implement
this without running a script at user login time (or more generally
when a session is established). We run our script through pam_exec in Ubuntu's
/etc/pam.d/common-session
:
session optional pam_exec.so type=open_session /path/to/script.sh
The 'optional
' bit here is really important. If you leave it out
and you ever have an error in your script, you will have just locked
everyone out of the machine (yourself included). As they say, ask
me how I know (fortunately I did this on a test virtual machine,
not a live server).
(Because we've found that limiting cron and at jobs to be necessary in our environment, we've also put this into /etc/pam.d/cron and /etc/pam.d/atd. This requires cron and at jobs to be in user sessions, but we were already doing that for other reasons.)
The script has to do two things. First, it has to turn on fair
share scheduling if it's not already
on. You have to check this on every session startup, because if all
existing user sessions go away (ie there's no one logged in and so
on), the whole fair share scheduling setup disappears. Because we
want to limit both CPU and memory usage, we set both 'CPUAccounting=true
'
and 'MemoryAccounting=true
' for user.slice
itself, all currently
existing 'user-*.slice
' slices, and all currently existing
'session-*.scope
' scopes. It's possible that some of this is
overkill.
Second, we set appropriate per-user limits (based on the various
bits of information about the size of the machine) by setting
appropriate 'CPUQuota=...%
' and 'MemoryLimit=...
' values on
'user-${PAM_UID}.slice
'. We also set a TasksMax
. As we currently
have our script implemented, it blindly overwrites any existing
settings for the user's slice any time the user starts a new session,
which has both advantages and drawbacks.
(All of this setting is done with 'systemctl --runtime set-property
'.)
We've chosen to not do any of this for sessions for system users,
including root. If the script sees that ${PAM_UID}
is outside
our regular user UID range, it does nothing, so root's logins are
unrestricted. We could have implemented this in the pam.d file
itself, using pam_succeed_if, but I feel that
scripts are a better place for conditional logic like this if
possible.
In the future, some of this may be possible to do through systemd
drop-ins for user.slice
and individual user slices. However, it
certainly won't be as flexible as you can be in a script, especially
if you want to behave differently for different UIDs and you have
enough users that you don't want to create and maintain individual
files for each of them. It would be nice to be able to reliably set
fair share scheduling once, though, and not have to keep re-setting
it through the script.
|
|