Wandering Thoughts archives

I'm giving up on upgrading my laptop from Fedora 14

I've recently reached a liberating decision: I'm giving up on any plans to upgrade my laptop from Fedora 14, unless something really surprising happens. I'll run it with Fedora 14 until the hardware dies or I have to replace it for some other reason and at that point, well, who knows what my options will be. The fundamental reason why is that I've decided I'm not willing to accept a less productive environment just in order to have an up to date OS.

(This sounds obvious in retrospect but for some reason I didn't even think about it this way until recently. There's a lot of my mind that thinks that upgrades are a good thing so they should be done even if they're a little painful.)

But this begs the question of what a productive environment is for me and why I can't get it on subsequent Fedora versions. What I've realized is that the core of a productive environment on my laptop is sshmenu plus a capable mini command applet or equivalent. Most of what I use my laptop for is logging on to other machines; I have a very well tuned environment for this today and I'm not interested in making it more awkward and less functional. There are sshmenu equivalents for Gnome 3 and other desktop environments but so far I've been unable to find anything equivalent to the Gnome 2 mini commander applet and my configuration.

(The crucial feature of the mini-commander is not 'a text entry box that will run commands'; it is 'a text entry box that I can use to ssh to hosts'. There are the former sort of applets for, eg, XFCE.)

With enough work, I could turn one of the alternate Fedora desktops into something that had the productive features I already have in my Fedora 14 Gnome 2 based environment. But if I'm going to do that sort of work I might as well go all the way and set up a laptop version of my much nicer custom environment (or perhaps something else nice). However, the whole point of using a standard desktop with minor customizations is to not go to that sort of work. I'm not interested in replacing a work-free current environment with a work-needed new one (not until I have no choice).

Coming to this decision feels oddly liberating, perhaps because now I can stop feeling somewhat guilty (and worried) that I've got such an old OS install that I've been 'too lazy' to upgrade.

As a corollary to this decision I'm giving up on spending more than a trivial amount of time looking into alternate desktop environments on Fedora 18 (or any future version of Fedora). There's no point in doing so unless I've got some reason to think that they now have the applets (or equivalents) that I want.

(This is the kind of entry that I write partly to argue myself into believing in it. And of course I'd be perfectly happy if a suitable environment fell into my lap but at this point I don't expect the necessary mini-commander applet to appear for any current desktop environment, Gnome 3 or otherwise, given that no one seems to have written and packaged one so far.)

MATE Desktop's failure in Fedora 18

For those of you who haven't heard, MATE Desktop promises to be the Gnome 2 desktop reborn. I very much like the Gnome 2 desktop for my midlevel machines and specifically for my laptop, which has been stuck at Fedora 14 for going on two years now precisely because I refuse to abandon a productive Gnome 2 environment. Since MATE is included in Fedora 18 I had high hopes of being able to upgrade at last. When Fedora 18 was initially released I eagerly spun it up on a virtual machine and installed MATE to try it out.

I don't know of a good way to put this: MATE Desktop in its current state in Fedora 18 is an abject failure because the MATE developers made an understandable but catastrophically bad decision.

You see, when the MATE Desktop describes itself as a fork of Gnome 2, it really means that. The developers did not take Gnome 2 and rebuild it; they took Gnome 2 and more or less changed everything that called itself 'gnome'. In particular they completely renamed the preferences system. MATE doesn't use Gnome 2 preferences; it uses MATE preferences, complete with a MATE preferences daemon. One immediate consequence of this is that you cannot use the very useful and important gconf-editor to edit MATE preferences. After all, gconf-editor edits Gnome preferences, not MATE preferences (perhaps you are starting to see the problems here).

(People who migrate to MATE may not immediately see the problem because MATE runs a one-time preferences cloning operation. That they need to do this should have been a big warning sign to the MATE developers.)

There are a whole lot of tools and programs that have been developed for Gnome 2 and now run under it (or Gnome 3). It is unlikely that many of them will be recompiled for MATE and the MATE libraries. How well any individual one is going to work in MATE is uncertain; at the least it's clear that there are going to be frictions, where you wind up with things like two preference systems running alongside each other. At the worst, some of the tools that I really want may simply not work because they have not been rebuilt and/or 'ported' to MATE (by running the equivalent of a sed pass over them to rename functions or whatever).

This practical issue is one part of why I abandoned MATE on the spot when I found this out. Another part is the mere existence of this issue demonstrates that the MATE people's priorities have nothing to do with mine. I want to continue using my perfectly functional Gnome 2 desktop environment. The MATE people apparently want to make a statement about how they are not Gnome. Well, congratulations. Consider it made and heard.

(Now I need a replacement laptop environment that I can stand, one that runs all of the magic programs necessary for a modern laptop that uses varied wireless, wired, and VPN connections, suspends itself at appropriate times, handles battery and power management, and all that jazz.)

The systemd dependency problem

When I wrote about what systemd got right I also mentioned in passing that it wasn't without flaws. It's time (and really past time) to start doing some elaboration on that, and I'm going to start with systemd's problem with documenting dependencies.

Systemd is fundamentally a dependency-based init mechanism; it starts things and orders startup based on what service needs what other service (among other things this determines what can start in parallel). All of this is well and good but it means that in order to write good systemd units you need documentation on how startup is structured, which is to say the standard dependencies involved. Your unit almost certainly can't run at arbitrary times, so in order to make it run at the right time (neither too early nor unnecessarily late) you need to know what to make it depend on. You may also need to know how to tell systemd that your unit provides a particular generic service like, for example, DNS lookups.

I'm going to be blunt: systemd in Fedora is falling down on this today. Part of this is systemd's fault and part of this is Fedora's fault (since at least part of how systemd units are ordered is up to the distribution and thus up to the distribution to document). Today it is fairly hard for a sysadmin who wants to write a good unit that works right to find out what they are supposed to depend on under what circumstances (and I speak from experience). Often it isn't obvious that you're missing something and that as a result your unit is only working through coincidence on your particular system (and may stop working right if something changes how systemd orders things).

(This problem is in fact so difficult that some Fedora-supplied packages get it wrong. For example their package for unbound does not assert that it provides DNS lookups, which means that even if other units properly say that they want DNS lookups they may well start before unbound does because systemd doesn't know any better.)

Systemd provides some documentation for this in the systemd.special(7) manpage but it's relatively sketchy and incomplete (from my perspective); it lists things but doesn't provide much guidance on how you want to structure your units and what you want to depend on in practice. Also, the section on depending on the network is actively enraging for many server administrators. It's very nice for systemd to tell developers that they should make their programs be nice and flexible in the face of networks appearing and disappearing, but they generally don't on servers and system administrators have to deal with many, many programs that have not been updated to how systemd wants them to behave. Worse, sometimes there is simply no rational way to do this sort of update.

(A related lack is that the systemd documentation does not clearly spell out how to tell it that your unit implements a particular target. Apparently the way to do this is to specify both Before=something.target and Wants=something.target; this is charmingly indirect, to put it one way.)

Related to this lack of documentation is a lack of tools for determining service dependency relations; systemd provides neither a 'what requires this' nor a 'what is required by this' query operation. Both are important in practice, especially if you're trying to audit your system to insure that its behavior is predictable and correct (ie, that you have the dependencies right so that everything deterministically starts when it should). Note that looking at the actual boot order is not sufficient for this because you don't know if the boot order is a product of actual dependencies or just how systemd decided to do things this time around.

(This collection of issues bit me in my recent upgrade to Fedora 18. Units that had been starting perfectly fine in Fedora 17 suddenly started not working; it turned out that they were missing dependencies and had only been working in Fedora 17 by coincidence. Trying to properly depend on DNS lookups being ready to go led me to discover the issue with unbound's own ordering.)

Related to this is the issue of missing dependencies. Systemd's selection of standard dependency targets and things that implement them is relatively sparse. Systemd provides neither tools nor good documentation for adding more, including targets that server administrators would like in practice. This is especially striking in the case of networking targets; if systemd is going to throw us to the wolves (as it does today), I would like it to provide some tools to at least help us implement our own meaningful targets (and yes, we're going to need them in practice). Even more useful would be standard fine-grained targets that systemd automatically notices and advertises (for example, 'IP address X has been assigned to an interface').

(I suspect that the best way to do this would be for systemd to support dependencies on DBus information, since I believe that such information is already broadcast across DBus for interested parties.)

Why .rpmnew files are evidence of packaging failures

The other day I tweeted:

Thesis: every .rpmsave file on a #Fedora system represents a packaging failure. The package should have provided a better override method.

(I made a mistake in this tweet; I actually meant .rpmnew files. Sadly I only noticed this mistake now.)

I should give at least one person a full explanation what I mean by this (and besides, I alluded to it last entry). In my traditional way, I'll start with a question: what creates .rpmnew files? I can think of at least three different causes:

• an upgrade of a multi-arch package that creates spurious .rpmnew files. I think that this can happen in some other situations with such shared configuration files.
• a package-supplied configuration file that then gets altered by other packages as they install or upgrade. If the original package comes out with a new version of the configuration file, you get a .rpmnew file that is basically useless.

• when the local sysadmin has had to modify a configuration file in order to properly customize or use the package. In other words, the sysadmin had to override the stock package configuration and had to do it by editing the package-provided configuration file.

The first two situations are clearly system problems. The first is a failure of the package management system itself; it should be smart enough to handle this instead of creating .rpmnew files that are exactly identical to their real version. The second is either a failure of the package or the package management system (depending on your perspective). Either way the resulting .rpmnew files are an abrogation of the system's package management responsibilities; the sysadmin made no changes but now has a mess to sort out (or more likely, ignore).

But the third situation is also a packaging failure. As I covered yesterday, the modern Linux world has developed plenty of mature techniques for configuring packages without ever having to touch package-supplied configuration files. Some of them take program changes (like support for including files or scanning directories), but those program changes are worthwhile by themselves and there are brute force techniques to deal with the situation even without program support. This is especially the case for anything that is commonly altered to configure (or reconfigure) the package; the more that the package configuration gets changed in real life, the more it should be set up this way.

(There are some package configuration files that are basically black boxes under almost all situations, where almost no sysadmin will ever touch them.)

This especially matters because packagers often understand much more about the configuration changes between package versions than most sysadmins will. Letting your package create .rpmnew files and then leaving it to sysadmins to find and merge changes is pushing the work from a few people who (should) understand the package well to a lot of people who don't necessarily (and who may not even notice the .rpmnew files for various reasons).

Debian shows how to do Apache configuration right (and Fedora fumbles it)

For reasons involving upgrading my office workstation to Fedora 18, I've had to deal with the stock Fedora Apache configuration setup for the first time in a long time. And oh boy, is it bad. It's a beautiful example of how not to do configuration files in a modern package manager world and is all the more shocking because I'm used to the Ubuntu Apache setup (which is the Debian Apache setup), which basically gets everything right.

Here is how Debian splits Apache configuration up:

• Apache modules put their configuration files in /etc/apache2/mods-available, but this doesn't enable them; there's a separate directory for that, /etc/apache2/mods-enabled. To enable a module you put appropriate symlinks in mods-enabled or, more likely, use a2enmod and a2dismod to do it for you.

• the configuration files for your websites are in /etc/apache2/sites-available; default is the default stock one (and default-ssl if applicable). However, once again, this doesn't enable them; that's /etc/apache2/sites-enabled, which has appropriate symlinks (managed by hand or with a2ensite and a2dissite as you prefer).

• configuration file snippets that don't otherwise fit go in /etc/apache2/conf.d. In a default configuration there is almost nothing in this directory that is active and most of it you don't want to touch.

• configuring the ports that Apache listens on has been split off into /etc/apache2/ports.conf (and this automatically handles SSL).

The remaining Apache configuration file is very short and consists mostly of comments; the Ubuntu 12.04 version has exactly 54 lines of configuration directives, most of them to set MPM parameters.

This is not perfectly modular, but in practice we can run a number of web servers, ranging from relatively simple to fairly complex, without needing to change any of the stock, package-provided files. The two most common changes (what your website setup actually is and controlling what Apache modules the server has enabled) both explicitly separate something simply being installed from it being enabled. Changing listening ports has been deliberately split off into its own file that has nothing else, so if you have to change it you still have a pristine main configuration file (that will keep being updated with package updates). You can remove or completely replace the default site just by removing and changing symlinks, while keeping the actual default around untouched for package updates (and to mine for ideas and so on).

Fedora, well, Fedora is not like this. It has no distinction between installed Apache modules and active Apache modules; if you have a module installed it drops a file in /etc/httpd/conf.modules.d and you can guess what happens next. It has no equivalent of even sites-enabled; your site is hardcoded in the main Apache configuration file along with a pile of other things (I'd call it 'your main site' except that Fedora doesn't even imagine that you might use virtual hosts). There is a directory for 'random configuration snippets', /etc/httpd/conf.d, but it's a complete mixed bag of actual general configuration directives and various packages adding Aliases and ScriptAliases to your site. It doesn't have any separation between installed and enabled either, and random Fedora packages feel free to drop bits in here (and in the process steal URLs out from underneath you).

All of this is catastrophically bad in the face of modern package management. In order to do anything with a Fedora Apache configuration you must change and delete files that are owned by various packages, leaving you to find and merge any future changes by hand (and you'd better also re-remove any files you removed when they're put back by package updates). This goes directly against years of experience of how to make things work well with package managers, and as Debian shows it's not as if Apache lacks the tools to create a genuinely modular and package manager friendly configuration. Nor is it as if Fedora is without experience at designing systems that work very well with modern package management.

(In practice the only thing you can really do with the Fedora Apache configuration is to burn it to the ground and start from scratch. Before I discovered the Debian way I had an approach for this, which I guess I get to repeat.)

There is a larger scale lesson for package management here that I encapsulated in a tweet, but explaining it properly is going to take another entry (this one is already long enough).

Turning off delays on failed password authentications

Today I got around to something on my office and home workstations that I should have done years ago: I turned off all delays after you mistype a password, both for ssh logins and for local things like su.

I've been an advocate against network authentication delays for quite some time. Over time I've come to realize that the same logic more or less applied to local authentication delays too. In theory they're there to slow down mass password guessing attacks, but in practice all they were doing was irritating me.

(They generally didn't slow me down because I was well trained that if su wasn't instantly successful, I'd mistyped the password and I should open up another 'su to root' window and do it again.)

As you'd expect (and hope), on Linux this is controlled through PAM. On Fedora 17, it's sufficient to change the 'auth' usage of the pam_unix.so module to have the 'nodelay' parameter; this tells it to not ask the whole PAM system for a standard delay if authentication fails. I had to change both /etc/pam.d/passwd-auth (apparently used by sshd) and /etc/pam.d/system-auth (used by su). A typical line is now:

auth   sufficient    pam_unix.so  nullok try_first_pass nodelay

(On my Fedora 17 machines both files have big warnings about their contents being autogenerated and they'll get overwritten by authconfig. Since I can't remember the last time that I ran authconfig, I didn't let this worry me.)

This gives you no delay at all. If you'd still like a little bit of delay you need to add a mention of the pam_faildelay.so module. I believe that it goes before pam_unix.so and it should look something like:

auth   optional     pam_faildelay.so delay=250000

(This delay is a quarter of a second. See the manpage.)

I haven't tested an Ubuntu system, but inspection shows that it does things a little bit differently. Based on looking at files, it appears that you want to modify /etc/pam.d/common-auth and then either remove the mention of pam_faildelay.so from /etc/pam.d/login or modify the delay time.

Having no delay on local password authentication is a potential security exposure to local users; it allows a local user to automate guessing attacks as fast as a program can run su, passwd, or the like. If this concerns you, use pam_faildelay.so to add a small delay; even a tenth of a second of delay will drastically slow down an attacker.

PS: my excuse for not doing anything about network authentication delays on my own systems for so long is that I just use SSH keys, so sshd almost never asks me for a password in the first place.