I've now disabled systemd-oomd on my Fedora desktops

December 5, 2022

Systemd-oomd is a somewhat controversial systemd component that, to quote its manpage, "uses cgroups-v2 and pressure stall information (PSI) to monitor and take corrective action before an OOM occurs in the kernel space". A while back, Fedora enabled systemd-oomd by default and set it to be applied to user@.service, the template for user slices. When I upgraded to the relevant Fedora version, I sort of shrugged and went along with this to see what happened. Nothing did for a long time, until I had a little incident:

Achievement unlocked: I just had systemd-oomd kill my entire X session for unclear reasons. Does it log details about the session it killed so I could see, say, which program was using too much memory? Nope.

Systemd-oomd has now been voted off my island.

First off, this is exactly how systemd-oomd is supposed to behave under memory pressure. The documentation is specific on this; systemd-oomd itself says:

[...] If the configured limits are exceeded, systemd-oomd will select a cgroup to terminate, and send SIGKILL to all processes in it. [...]

By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes (across all of your sessions, if you have more than one). This may make sense for a multi-user system or a system running important daemons in addition to user sessions, but in my opinion it doesn't make much sense for a desktop that's only running a single user's session.

The actual user experience when this happens is also terrible. Unlike the kernel OOM killer, systemd-oomd doesn't log any details and makes no attempt to identify which process, sub-cgroup, or whatever is actually causing problems. If you are lucky, you will already have some idea (perhaps because things have happened slowly); if you're unlucky, as I was, you're doing things, something quietly starts running away without any obvious symptoms that it's eating memory, and the first you know is that you're staring at a login prompt.

By contrast, the kernel OOM killer acts later but would have only killed a single process (almost certainly the one at fault, in my case) and in any case would have left copious information in the kernel logs. I would much rather have dealt with the kernel OOM killer in this case; even if it had freaked out and killed my entire session, I would know a lot more about what had gone wrong.

It's possible that someday we'll wind up with systemd-oomd on our Ubuntu servers (it's not there in our current 22.04 configuration). It may be more tolerable there, but even then we may prefer to put hard memory limits on people's user slices and then rely on the kernel OOM killer acting at the cgroup level. If nothing else, this gives us better logging about what happened (and probably not terminate the entire session or worse, all of their login sessions).

Comments on this page:

The problem that it kill all processes in a cgroup. The solution is to be configuring the cgroups and have each app/service in a cgroup with correct config. To this you effectively need to run the app as systemd user service. Only gnome and kde have the configuration that does this cgroup stuff. In fact the cgroups where useful on the server has I was able to constrain ram too.

By AnotherUnixHerder at 2022-12-08 04:43:11:

You should try this one some time https://github.com/hakavlad/nohang. Have been using it before systemd-oomd existed.

Written on 05 December 2022.
« How to lose some of your tabs in Firefox 107+ (and possibly earlier)
Why being able to partially distrust a Certificate Authority is good »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Dec 5 21:05:54 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.