Wandering Thoughts archives

cgroups: so close and yet so far away from per-user fair scheduling

Suppose, not entirely hypothetically, that you have some shared, multiuser compute servers. Further suppose that sometimes, person A is running a single compute process while person B is running, oh, nine. Since Linux divides CPU time among processes without caring who owns them, this means that person A is getting 1/10th of the CPU while person B is getting 9/10ths of it. This doesn't seem entirely fair; it would be better if A and B split the CPU 50/50 regardless of how many compute jobs each ran.

(Since these are all multi-CPU machines, the real examples are more complicated. But yes, periodically there are more compute jobs than there are cores.)

Modern Linux kernels come with support for cgroups, which is designed to enable this sort of stuff. I'll cut to the chase: cgroups can at least in theory do exactly the per-user fair scheduling that we want here. In practice Linux is let down by the current state of the user tools, which lack the features you need to make this feasible.

How to use cgroups to create per-user fair scheduling is pretty straightforward; you just put each user into their own cgroup (or at least each real user, you might want to do something different with system daemon UIDs) and give each user cgroup the same cpu.shares value. The system will then evenly divide the available CPU up between all users with active processes. The obvious place to manage all of this is in a PAM module, which can create the per-user cgroup on the fly the first time it's necessary and so on.

The kernel support for all of this is there, as is most of the user level tools you'd need (in the form of libcg and associated programs); there's even a PAM module, which classifies users into cgroups based on a configuration file or two. However, what the tools don't have is any ability to have generic entries in the configuration files for creating cgroups and assigning users to them. If you want to have one cgroup per user, you get to write them out explicitly (and then the tools will create them all ahead of time). Oh sure, you can generate the config files with a script, but you also have to poke various daemons every time you want your config file changes to take effect. Things get annoying fast.

(I also wonder how happy the kernel will be to have a thousand or so cgroups, almost all of which are unused at any given time (given that only a handful of our users will log on to a compute server at once).)

PS: the tragic thing is that a hard-coded PAM module would be almost trivial (and I've written PAM modules before). But that would mean building and maintaining a custom PAM module, and this issue is not quite important enough here to justify that.

(Like most sysadmins, we get a modest amount of hives at locally developed software. The closer we can be to stock systems the happier we are, because it means that someone else is maintaining the software.)

Sidebar: systemd, cgroups, and per-user fair scheduling

It appears that this entire issue will be rendered moot for us if and when Ubuntu does an LTS release that's based on systemd. Per this blog posting and more documentation, current versions of systemd can already put users into per-user cgroups for you with the right options set on the systemd PAM module.

I admit that I'm kind of looking forward to this.

(Well, I'm not looking forward to yet another init replacement and init system, but there doesn't seem to be anything I can do about that. Maybe systemd will be the last one for a while.)

My three sorts of (Linux) desktops

To kind of expand on a comment I made on an earlier entry, I broadly have three sorts of Linux desktops.

On machines I don't use very much and thus don't really care about I use whatever the default environment is, sometimes slightly customized if it's easy enough and I'm using the system more than once. All I really care about on these machines is getting to a terminal emulator (so that I can run ssh) and sometimes being able to run a browser. Oh, and actually connecting to the network.

On machines that I use enough to want to be productive on, I make some or many of my standard Gnome customizations (also). I still only really care about ssh and a browser, but now I want the whole process to be convenient and not too annoying.

(This assumes that these machines run Gnome as their standard environment. My EEEPc didn't, so I wound up customizing its environment instead of trying to build a Gnome environment on it. My general attitude on these machines is that it's not worth fighting the defaults, but it is worth customizing things.)

On my primary machines, the machines I sit in front of all of the time (either at work or at home), I run my full custom desktop environment. My desktop is a lot of work to set up but it's what I'm used to and it's highly tuned to work just the way I want it to. This environment has basically nothing to do with Gnome except sometimes using some Gnome applets or running some Gnome programs to do specific things.

(There's a very hard line between the second and the third sort of desktop, but the line between the first and the second is of course fuzzy.)

Since I only use Gnome on machines that I don't really use that often or care that much about, I'm pretty indifferent to what horrors the Gnome developers do to the Gnome environment provided that they don't hide the terminal application or the web browser (or make them very inconvenient to use). Gnome could make itself very irritating on the second sort of machine by eliminating both the sshmenu applet (which Gnome 3 sort of has) and the mini-commander (which it doesn't seem to have yet), but this wouldn't necessarily be fatal for my ability to use Gnome productively; I could live with starting shells and typing 'ssh whatever' myself if I had to.

Sidebar: the other reason to be lazy about Gnome on laptops

My strong impression is that an ever-increasing amount of things that you need to work on a laptop basically require a full Gnome environment (or far too much hacking). I'm thinking of things like useful wireless networking, dynamic networking in general, and automatically suspending and unsuspending as the lid is closed and opened. Letting Gnome worry about all of this magic is certainly the easy approach to dealing with it.

It is thus not terribly surprising that my work laptop runs Gnome. My time is too limited to spend it working out how to do wireless by hand.

(Working out how to do dynamic mounting of removable media was annoying enough.)

A package dependency failure in Fedora 15

I have a quite simple view of package dependency failures for packages in a distribution's official repositories: they shouldn't exist. More precisely, any package with unmet dependencies should be excluded from the repository. Letting such packages sit in your repository is an unhelpful lie, one that particularly irritates sysadmins, and in my opinion it makes you look just as amateurish or at least frustrating, as frustrating as those stores that list things as available only to turn out to be out of stock on them when you actually try to order.

(Ideally the repository would exclude all uninstallable packages, but there are twisty situations where checking that is complicated. But checking for unmet dependencies is relatively simple and straightforward.)

Today's offender is Fedora 15 and the gnome-applet-sshmenu package (I've grumped at Ubuntu for similar issues, so turnabout is fair play). As covered in Fedora bug #711555, this depends on a package capability (Ruby bindings for the Gnome panel) that is not available in Fedora 15 because of the upgrade to Gnome 3. This unmet dependency is trivial to find and determine. But it wasn't, and so you have an uninstallable package sitting in the Fedora 15 repository, tantalizing and annoying people.

Fedora should have detected this and at least rejected the binary package, well, excluded it from the repository, in just the same way that it would reject any other flawed binary package. (Rejecting the source package that it was built from and all of the binary packages that the package generates would attract the maintainer's attention, but might be too drastic.)

There are various side effects of this broken package sitting around in the repositories. The one that I ran into is that you can't do a yum upgrade from Fedora 14 if you have the package installed; if you try, you get various sorts of odd errors that take some work to sort out.

(Another yum upgrade failure appears to be due to the presence of libpanelappletmm, but that seems to be a different issue.)

In other news, it looks like I need to learn how to write Gnome panel applets in Python if I want to upgrade to Fedora 15 any time soon.

(The prospects for Ruby bindings for Gnome 3 appear low, and with no Ruby bindings, so much for the sshmenu applet that's an important part of my nice ssh environment for Gnome. Fortunately I can probably do a hack job replacement in Python without too much work, assuming that the Python bindings are sane.)

Sidebar: why detecting uninstallable packages is hard

Suppose you have a package A that depends on B and C, both of which exist and are installable separately. A simple dependency check will pass package A and say it's installable. Now, suppose that B and C conflict with each other somehow so that you can only have one of them installed at the same time. A is no longer installable, but to detect this you must detect that B and C clash. Obviously, the fully general version of this problem can have the conflict occurring down in the depths of a dependency chain instead of in direct dependencies.

(RPM makes the problem worse by increasing the number of ways that packages can clash; for example, they can try to install two different versions of the same file. So in an RPM based system it isn't sufficient to just check for explicitly declared conflicting packages.)

I'm sure that this is solvable, but it's nowhere near as easy and simple as detecting completely missing dependencies.