Wandering Thoughts archives

Remote applications and Gnome settings: an irritation

Here is a conundrum that I have recently run into, posed as a question: when you run a Gnome application remotely through ssh-forwarded X, where does it get its Gnome configuration settings from?

In the old days of X resources, this sort of question had a simple answer; custom settings were attached to the screen itself (they got stuffed into an X property) and thus were accessible to all X programs, regardless of where they were running. In the new ultra-modern world of GConf and friends, the answer is far from obvious.

In theory, Gnome applications get their settings from a per-user GConf daemon, which they talk to via DBus. As usually implemented, DBus is strictly local to the local machine, so the answer should be that they get their configuration settings from whatever Gnome settings you have established on whatever machine the application is running on. (I will not think too hard about how the necessary user GConf daemon gets started; I find it better not to think much about Gnome magic.)

In practice, this seems to work for some settings but not others. In my testing, Gnome's application specific settings work on a remote machine, but the general desktop settings do not (even when they influence application behavior). Specifically, I know that a custom setting for the cursor_blink key in /desktop/gnome/interface is ignored when my session isn't actually on the machine that I'm running gnome-terminal on. Sadly, this is is kind of a problem for me.

(In theory this behavior could be justified on the grounds that my session isn't on that machine, but it results in undesirable behavior that I have no way to control. And yes, gconftool on the remote machine claims that the key is properly set and everything works if my session actually is native to the remote machine (apart from the bit where I am using a slow LTSP instead of my nice desktop machine).)

The problem with overly verbose package installation

We just applied a kernel update to all of our Ubuntu LTS machines. This is a quite verbose process; on the machine I captured logs for, it produces no less than 40 lines of status output (and another 14 lines before things start installing). And this count understates things because some of those are very long lines that wrap around on any real terminal.

(The kernel updating was done with 'apt-get install ...'.)

I'm sure that the authors of the Ubuntu kernel package and all of the management programs involved in this update process thought that they had very important things to tell people, and some of this is because we have a fair number of kernels installed as a result of running these systems for a while; we might get about 14 lines less if we went around removing old kernels by hand. Sadly, all of those people are wrong, and I will tell you why: I lied about the line count.

On a few of our machines, including the one I have logs for, applying the kernel updates didn't produce 40 lines of output; it produced 41. The extra line in the middle was an error report (possibly not a significant one; it's not clear and the machines all rebooted with the new kernel). How many machines? I don't know. The difference between 40 lines and 41 lines is not very easy to notice, especially in repetitive text that you see ten or twenty copies of.

And this is the problem: overly verbose package installation hides problems. Problem reports are a needle in a haystack, and the haystack is your regular output; the more regular output, the less the needle stands out. Produce almost no output and the needle is glaringly obvious.

The other factor is how simple and patterned your regular output is. Yum has a fair bit of output by default but it has a very simple pattern (it's progress bar after progress bar), so it is relatively easy to pick out exceptional messages (not as easy as it would be if there was less output, though). The Ubuntu kernel update process's regular output is unpatterned, normal text, so a problem message does not particularly stand out unless you look carefully.

(By 'patterned' here I mean the abstract visual patterns, divorced of particular words and so on. Humans are quite good at spotting breaks in regular visual patterns; we are much less good at spotting particular things in a random mash of visual stuff.)

It's my jaundiced opinion that the problem here is a cultural one. I doubt there's anything intrinsic about apt-get that requires things to be verbose; instead, Debian and now Ubuntu has created a culture where it is both acceptable and routine for package installation to produce a lot of status messages and reports about completely routine events. At this point, the keepers of the culture might even view it as somehow wrong for packages to be silent.

(By contrast, the culture surrounding RPMs has generally been strongly in the 'no noise is good news' camp, where RPMs should not generate output during installation, removal, and upgrades unless something goes wrong.)

Sidebar: the particular error we saw

For the benefit of anyone who is curious, we saw error messages of the form:

Setting up linux-image-2.6.15-55-amd64-server (2.6.15-55.83) ...
ln: creating symbolic link `/tmp/mkinitramfs_8tcKVS/bin/true' to `/usr/lib/klibc/bin/true': File exists

I don't know what causes these. Since the machines did reboot into the new kernel and do not seem to have exploded, my motivation for digging into the depths of potential faults in initial ramdisk building is low.

Why the pam_mail PAM module is not my friend

Traditionally, one of the things that the standard login program did when you logged in was to look at the state of your mailbox and report 'you have (new) mail' if this was the case. This worked well when everyone logged in to Unix systems on serial terminals, sort of well when people used telnet and rlogin (both of which actually run login), and went downhill when SSH became common, because SSH does not run login at all.

Clearly the right solution (in the PAM world) was a PAM module to look up this information and report it. Hence the pam_mail module, which is present on many Linux distributions and used by default on at least Ubuntu. (Ubuntu uses it in the PAM configurations for login, sshd, and su, where it is only used to set $MAIL.)

This sounds great, but there's two problems with pam_mail. First, it always attempts to stat /var/mail/<user>; there is no way to turn it off on a per-user basis. If the module is configured, everyone gets this service (this is unlike login, which traditionally respects a $HOME/.hushlogin file). Second, it doesn't just do this when you log in via SSH, which is what you might expect; instead, it does this any time you authenticate to sshd and do something. In particular, running 'ssh host command' will cause pam_mail to try to stat /var/mail/<user> (even though you'll never see the message).

Now consider what happens when access to /var/mail stalls; perhaps it is on a dying disk, or perhaps it's NFS mounted from an NFS server that is having problems. Not only can you not log in to the machine via SSH, you can't even use SSH to run commands on the machine (whether as root or any other user). And all of this is because of an innocent sounding PAM module buried in a PAM configuration file.

Our /var/mail is NFS mounted from an NFS server. You can probably see where this one is going; today, that NFS server went down and havoc ensued. Thus, pam_mail is very much not my friend right now.

(It's also no longer present in /etc/pam.d/sshd on our Ubuntu machines. We're going to tell users who really care about this to put something in their personal dotfiles.)