Wandering Thoughts archives

Illustrating the Ubuntu clown car, AccountsService edition

AccountsService is a freedesktop.org thing to let other programs get (and set, with appropriate magic PolicyKit permissions) various information about user accounts over DBus (because everything has to go over DBus these days). The package's major component is accounts-daemon, which implements said DBus service. Ubuntu 12.04 ships with a modified version of accountsservice 0.6.15.

(By the way, it also theoretically lets other programs create, delete, and modify system properties of logins. If this is not alarming you, you are probably not a sysadmin.)

As shipped by freedesktop.org, accounts-daemon already has one problem; whenever the daemon starts up, it scans almost all accounts in /etc/passwd and gets the (full) group membership list for each of them, rereading and reparsing /etc/group for each account. Large sites have serious problems with this (also) (this bug report was found by my co-worker). Fortunately for us we only have a 500 line /etc/group and an 1800 line /etc/passwd, and it turns out that on a modern machine parsing a 500-line file over 1700 times is barely noticeable.

(This problem is particularly striking because it isn't necessary. Accounts-daemon is doing all of this work just to see if an account is in group wheel, and for that there is a much faster check that does not need to find the account's full group list.)

But this wasn't good enough for Ubuntu. Two of the account properties that you can get or set are the account's language and its locale (okay, the Ubuntu version adds the locale). In the stock code, these have no default value; if they have not been set explicitly over DBus by an outside program, they're null. In Ubuntu 12.04, Ubuntu decided that this wasn't good enough. If either is unset and you ask for its value the Ubuntu code tries to guess their correct value by opening and crudely scanning ~<user>/.profile (assuming that ~<user>/.pam_environment doesn't exist) and then falls back to /etc/default/locale. Once the code finds a value (from either source), it carefully validates the value by running an external program; in the case of the user's language, this external program is a shell script that runs a Perl program (among other things).

(See /usr/share/language-tools/language-validate. I am not making this up.)

By the way, none of this is cached. For example, the code is perfectly happy to read /etc/default/locale (and then validate the result) several thousand times.

Let's set aside everything that's wrong about crudely, unconditionally, and blindly scanning a .profile (especially since the code to do it is very limited and seems broken), because it gets worse. Ubuntu has set up its Unity environment such that the program responsible for the Unity tools and logout menu asks accounts-daemon for the locale and language of every known user when you log in (and possibly at other times too). Let me assure you that opening and reading (almost) every user's .profile (twice) takes a noticeable amount of time even with only 1700 or so real users; in fact in our fileserver environment it generally takes at least several minutes. Under some circumstances it also puts a visible load on the system (and eventually the NFS servers).

(Oh, and until this scan finishes the tools and logout menu is empty. This is somewhere between annoying and disconcerting to users, and leaves them unable to log out cleanly.)

While the same modifications to the accountsservice source package are in current versions of Debian testing, it's clear that they come from Ubuntu and were propagated into Debian. This really is an all-Ubuntu show; these modifications aren't in the upstream code and Ubuntu did not get them from someone else, they developed all of this mess themselves.

Sidebar: how bad the scan of .profile is

If you have the Ubuntu 12.04 source package for accountsservice unpacked somewhere, look at src/user.c's user_get_profile_env function. Simplified and with a bit of pseudo-code, this does:

char line[50];

fp = fopen(profile_path, "r");

while ((fgets(line, 50, fp)) != NULL) {
  if (line starts with 'export LANGUAGE="') {
    ... extract the value
  }
  if (line starts with 'export LANG="') {
    ... extract the value
  }
}

I wish I was making this up. In low level mechanical flaws, I'm pretty convinced that this potentially breaks in interesting ways if you have lines in your .profile that are longer than 49 characters and it ignores any 'export ...' lines that are indented.

Another problem with how Debian builds from source packages

When I wrote my first entry on this I didn't realize one more significant problem with the way Debian builds packages from source (because I hadn't run into it my own package building). This is that Debian builds binary packages in the source package working area.

It's easy to see how this came about. Since the unpacked, working form of a Debian source package is a directory tree with the source code and some extra information, doing an in-place build when compiling and building binary packages is both trivial and the obvious and simple approach. And usually everything works fine.

The problem is that this has the same failure mode as patching your source code in place during the build. You are crucially dependent on the software's build and cleanup processes actually cleaning everything up, both routinely and especially when things go wrong. If things do not completely clean up, always, then your working area for the actual package gets contaminated with artifacts of the build process, artifacts which you will have to disentangle from your own changes to the package and then remove by hand. You're also dependent on the build process not modifying anything in the original source tree; if it makes modifications itself, you're again going to have to disentangle and revert things by hand.

But wait, we're not done. There's another worry: reproduceable builds. Suppose that you are repeatedly building versions of the package (for example, you're testing a change). How sure are you that the build process has not contaminated the source tree such that your repeated rebuilds are producing a different result than what you'd get if you started from a just-unpacked tree? After all, the same source tree is being reused over and over again for each rebuild.

(In theory the package's cleanup process is not supposed to leave anything behind. Practice may be different from theory.)

Contrast this to the situation with a separate build tree. When you use a separate build tree, you can clean it up simply by deleting it all and you can always start from a known state by recreating it every time you build the package. The software being built can do whatever bizarre things it wants to inside its source tree (including patching its source code on the fly) and it won't contaminate anything else or get confused with your changes.

(The downside to a separate build tree is that you have to set it up every time. This slows down building packages as you copy and/or extract a potentially large source tree into your new build tree.)

Why I don't like the Debian source package format

I'm about to start modifying an Ubuntu package to build a modified version, so I've been reminded of all of the reasons that I don't really like the way Debian source packages work (Ubuntu uses the Debian package format). To explain why, I need to start with a brief description of how Debian source packages are put together.

Like all source package formats, Debian source packages need to contain the four essential things, but they do it in an odd way. Source packages generally come in two different forms; let us call these the distribution form, which is what you download, and the working form, which is what you expand the distribution form into. The distribution form of a Debian source package has three components: a .dsc file of basic metadata, one or more bundles (usually one as a compressed tar archive) of the upstream source code, and a bundle of Debian modifications that are applied on top of it. The working form of Debian source packages is a directory tree of the source code, which is created by unpacking the upstream source code and then applying the Debian modifications to it; after this, the source tree will have a debian/ subdirectory with control files that describe how to build and package the source.

The largest problem with this is that the working form of Debian source packages has the upstream source already modified. The upstream source is not patched as part of building the binary packages (well, mostly); the upstream source is patched the moment that you even look at the source package. A direct consequence of this is that the control files do not exist until after you have patched the upstream source, or at least they do not exist in any readily accessible format.

Now we get to the mess that is the bundle of Debian changes. Debian source packages have two formats for this (or at least two dominant, non-experimental formats); the changes can be either a single giant patch file (which must include creating the debian/ subdirectory) or a tarball with the Debian control files and a set of quilt-based patch files (cf). The quilt-based format is sane, but it's the newer one. The giant patch file format is the older one and still quite common, but there are two things wrong with it.

The first thing wrong with it is that it's terrible as a format for modifications, especially for sysadmins who want to add a change to an existing Debian or Ubuntu package. Let me count the way:

• a single giant diff is hard to read, especially when it also includes creating the control files in the debian/ directory.

• a single giant diff smashes logically separate changes together into one big mess.

• it's impossible to separate out your own changes so that you can easily apply them on top of the next Debian or Ubuntu package update; you're better off keeping diffs of your changes outside of the source package system and then (re)applying them by hand.

The second thing wrong with it is that all of its problems spawned a bunch of workarounds, which basically use quilt or something like it in the debian control files to apply changes as the software is built. The large scale consequence of this is unpredictability. When you get and set up a Debian source package as a non-specialist, the resulting source tree may or may not reflect the Debian modifications and there is no single procedure that you can follow to make changes to it (especially maintainable changes).

(A meta-consequence of the introduction of a new and radically different format is that there are plenty of guides on 'how to do things with Debian source packages' out there on the web that hasn't been updated to discuss the new format. This is again a problem for non-specialists, who don't know enough to know that the guide is incomplete until something goes wrong. Note that I am barely more than a non-specialist (and most of that is due to writing this entry).)

A smaller irritation but something that is symptomatic of how Debian does things is how you change the build version of generated packages. A normal, sane packaging system would have an explicit field for this in metadata somewhere. In the Debian package format, the build version is derived by implication: each entry in the Debian changelog for the package (found in the debian/ subdirectory) has a build version attached to it, so the build system 'simply' finds the build version of the top (most recent) changelog entry and uses that.

(On reflection, I've decided not to say anything about the debian/rules file. I'm a non-expert in working with Debian source packages and so I suspect that everything about it is much clearer and more obvious to people who are Debian package building experts. Every source package system involves a certain amount of arcana and doing the actual compiling and so on is often where it's concentrated.)

How not to write kernel messages

Suppose that you reboot your Ubuntu 12.04 Linux machine and see, among the kernel boot messages, the following:

Yama: becoming mindful.

Perhaps you've heard something about Yama and would like to understand what this means, perhaps you're looking for boot anomalies in general because of an odd problem you're having, or even perhaps you'd just like to understand your kernel's boot sequence. Is this message helpful?

Of course not. For most people this is completely opaque and simply not a useful message. Even if you've got some idea of what Yama is, the best you can do is to take a guess at what this message probably means. A much better version of this message would be something like 'Yama LSM initializing', and an even better version would change the code so that it only appears after the Yama LSM actually is initialized.

(Printing this message early on is probably excusable; at least in the current Linux kernel, the Yama LSM panics the system if it can't complete the remaining initialization. I assume that the registration functions it calls are not supposed to fail, even though they can return errors.)

Linux is far from the only Unix to do this, but for various reasons the Linux kernel has taken it to new heights (or depths). Quite a few kernel messages are not written for outside people, and in fact they're often not even written for general kernel programmers; they're pretty much written for the people who are working on the particular subsystem, the sort of people who know off the top of their head what Yama is and what an in-joke message like 'Yama: becoming mindful' actually means.

I'm not advocating for a totally buttoned down style of messages and sometimes you really do need debugging messages that require so much domain expertise that there's no point in making them comprehensible to outsiders. But most of the time kernel messages don't have to be anywhere near as obscure as they are.

(I expect that this is also the kind of thing that you generally can't fix by submitting a patch, because a patch to make a kernel message 'more boring' is highly likely to be politically explosive. Changing this needs to come from a cultural shift and internal pressure, not attempts from outsiders.)

Ubuntu 12.04 and symbolic links in world-writeable sticky-bitted directories

We're in the process of upgrading from Ubuntu 10.04 to Ubuntu 12.04, and today we ran into a serious surprise. If you have a symlink in a world writeable directory that has the sticky bit set so that only the owner of a file can delete it (for example, /tmp), only the owner of a symlink can dereference it. Everyone else will get EACCES on any operation that attempts to do so, including things like attempting to stat() the symlink. If this is happening to you, your Ubuntu kernel will log messages like:

non-matching-uid symlink following attempted in sticky world-writable directory by cat (fsuid 915 != 2315)
yama_inode_follow_link: 16 callbacks suppressed

(If you are testing this, note that the stat program won't trip over it because stat uses the lstat64() system call to look at the symlink itself. cat will fail, as will things like 'test -f <symlink>'.)

This change has alleged security benefits (cf). It also caused an important part of our mail environment to explode, and it is not compatible with historical behavior (both for Linux and for Unix in general). Fortunately you can turn it off; in the Ubuntu 12.04 kernel you need to set /proc/sys/kernel/yama/protected_sticky_symlinks to 0 (instead of the default 1).

(This is also the sysctl kernel.yama.protected_sticky_symlinks.)

The history of this is somewhat tangled. The Linux kernel has a generic idea of 'security modules' (called LSMs) that are used to implement additional security policies over top of the standard Unix permissions; SELinux is the best known LSM, but there are others. In Ubuntu 12.04, Ubuntu has included their own 'Yama' LSM and used it to implement this restriction; a version of Yama that has this was proposed by Ubuntu's Kees Cook as far back as 2010 (also). The standard kernel code also has a (or the) Yama LSM, added at the end of 2011, but it lacks this restriction (and seems to never have had it).

(We only work with Ubuntu LTS releases, so I don't know if Yama and its restrictions appeared in any non-LTS releases between 10.04 (which definitely doesn't have it) and 12.04, or if it's genuinely new with 12.04.)

However, Kees Cook also proposed this restriction as a general patch at the end of 2011. A version of this finally made it into the kernel on July 25th (after 3.5 was released but before 3.6, so this restriction will be in 3.6 when it's released). The actual restrictions are I believe only slightly different (the official kernel code allows the symlink to be followed if it's owned by the owner of the directory), but the important thing that the sysctl to control it has changed. In the official kernel this is controlled by /proc/sys/fs/protected_symlinks (aka fs.protected_symlinks as a sysctl).

(Credit where credit is due: a bunch of these links were found by one of my co-workers while I was busy doing less productive troubleshooting.)

I assume that Ubuntu will drop this restriction from their version of Yama and rely on the standard kernel code when they eventually do an Ubuntu release with a 3.6 or later kernel. At that point, anyone who is turning it off will need to change their sysctl. Hopefully we will remember all of this in a couple of years when the next Ubuntu LTS release happens and this becomes relevant to us again.

I am a grumpy Unix oldtimer and a sysadmin so my opinion on this change is very low. I found especially laughable the idea that Linux kernel people could confidently assert that there was no (legitimate) application that would be affected by this change. Congratulations, you blew up part of our mail system; I guess it doesn't exist.

(Our mail system works in such a way that I believe there are no security risks for us here, but justifying that will take an entire entry.)

Sidebar: another restriction, on hardlinking

Both the Ubuntu Yama LSM and the new standard kernel code for this stuff include a second restriction, this time on what can be hardlinked. In the standard kernel, this is controlled by fs.protected_hardlinks and blocks hardlinks to devices, setuid files, executable setgid files, and files that the UID making the link cannot read or write to (except that the owner of a file is always allowed to make hardlinks to it). Note that this blocks hardlinks to other people's world-readable files if you can't write to them.

The Ubuntu Yama LSM has a similar sysctl in its /proc directory and I suspect that its restrictions are much the same. On Ubuntu 12.04, if this restriction is triggered the kernel will log the message:

non-accessible hardlink creation was attempted by: ln (fsuid 2315)

Since I've just discovered this restriction, I don't know if we'll wind up turning it off on our 12.04 machines.

Linux ps's problem with login names

I generally like the usual Linux verions of various utilities, GNUisms and all; at least they work reliably under many circumstances. The Linux version of ps is not one of those programs. Unfortunately, for reasons that I believe are at least partly political, Linux ps is a mess; as one example, the major distributions have had to fork procps (and the original main code hasn't seen a release for several years).

One irritating example of this mess is how ps treats login names, specifically how it treats long login names. To put it simply, unless you go to a lot of work ps prints all login names that are longer than eight characters as numeric UIDs. This is what you'd call a problematic limitation because Linux distributions have been using long login names for system accounts for years and running daemons as some of those accounts. In other words this limitation triggers all the time on modern Linux systems when you're looking at all processes (as sysadmins do with reasonable frequency), even if you don't have any users with long login names. (If you do, well, things are worse.)

In a sane world ps would have a simple command line switch to override this behavior. In this world ps has almost every option under the sun but this one (as you can see if you browse the manual page); as far as I know, you will search in vain among all of the options for 'long' and 'full' output and so on. The only way to change this is to resort to specifying a custom output format with a magic modifier on the login name field.

(Just as irritating is where the limitation and workaround are documented. The general existence of the limitation is mentioned in a single sentence at the end of the NOTES section of the manpage. How to work around it is sort of covered in the writeup of the -o option, which discusses how to widen fields. The actual size limit for login names is not covered anywhere; you have to intuit it from knowing that login names traditionally had to be eight characters or less.)

Sidebar: An example of how to work around this

Taken from a script I have that uses ps:

ps -A -o user:17= -o pid= -o ppid= -o tty= -o args=

This shows how to combine omitting column headers with force-widening the field. Change 17 to some number that makes you happy.

If you want to see how long your login names are:

cut -d: -f1 /etc/passwd | awk 'length($1) > 8 {print length($1), $1}' | sort -nr

On some of our Ubuntu 12.04 machines (installed with Gnome stuff), the longest system login names are speech-dispatcher, avahi-autoipd, Debian-exim, and messagebus. The latter two are used by common daemons (Exim and dbus-daemon); I don't know if the first two are or if they're just used to own files.