Wandering Thoughts archives

How not to issue Linux kernel security update notices

This is a rant.

First, do not include useful information about how readily exploitable the bugs are. It is very well to tell people what subsystem the bug is in, but that's not enough. If there is a bug in the OSS code, can that be exploited from user level even on machines without any active sound hardware? If there is a bug in an obscure network protocol that the machine isn't using, is it still accessible to a user?

Relatedly, when you write that a bug is potentially exploitable to compromise the system, how strong is the potential? There is a not insignificant difference between 'this is likely exploitable but no one has bothered to write one yet' and 'in theoretical circumstances, with the right tailwind, you might be able to use this'.

This information is almost always available out there on the Internet (and your own security team should be using it to assess the urgency of the update), but feel free to make busy, harried sysadmins around the world redo all of this research. We love that.

Next, when you write your announcement and are describing what it fixes, include all of the security issues fixed since the beginning of time or at least the initial release of this version of your distribution, even if you first fixed them in a prior kernel update that you released months ago. On no account mark these old fixes in any particular way, so that sysadmins can tell which issues are new and which issues they are already not vulnerable to.

(Speaking from personal experience, sysadmins love being driven into alarm by an exploitable security issue that it turns out that they fixed months ago. Especially on a Friday afternoon. No, really!)

To make this more awesome, release a stream of separate kernel updates for specialized versions of your kernel (or different versions of your distribution) and make almost all of the text of the update message exactly the same as your normal message. Sysadmins will readily notice the differences between the following two Subject headers (and know what they mean and which ones matter):

Subject: [USN-1092-1] Linux Kernel vulnerabilities
Subject: [USN-1093-1] Linux Kernel vulnerabilities (Marvell Dove)

And they will certainly not waste their time reading very carefully through your list of fixed bugs trying to find any new important issues that it turns out wouldn't apply to them anyways because they don't use the specialized kernel package and besides, it doesn't even run on their architecture.

(For bonus fun you can still include security bugs that do not at all apply to the architecture that this version of the kernel runs on, like x86 specific bugs for an ARM kernel.)

Finally, be inconsistent in all of this. Force people to pay close attention to things like who wrote this particular kernel update message and guess what it means, in an attempt to pick out some pattern of how they need to read any particular update message.

If you're doing all of this, I'm really not sure why you're bothering to put details into your kernel security update messages in the first place. Just replace the entire body text with 'there is a new kernel security update that may apply to you. Type <appropriate package manager commands> and reboot your machine if prompted to'. The useful information content will be almost the same, you'll save time, and sysadmins will be marginally less annoyed with you because at least you're being straightforwardly useless.

(PS: yes, this is Ubuntu. And it's yet another entry in the 'why Ubuntu is not my favorite Linux distribution' category.)

How to configure a (PPPoE) DSL connection on a normal Fedora 14 Gnome desktop

The first and most important thing to know is that you should completely ignore what you'd think is the right way to do this. Trying to use what you find at System → Administration → Network for this (if there's anything there) is somewhere between useless and actively counterproductive, even though it can be used for at least some other network configuration things.

(This is system-config-network. If you installed Fedora 14 from scratch instead of upgrading from a previous Fedora version you may not have it installed, in which case I strongly recommend that you keep it that way.)

Instead what you want is hiding inside the network manager applet (nm-applet). Click on the applet, then VPN Connections → Configure VPN. Yes, really. You will get a tabbed widget open to the VPN tab; switch to the DSL tab instead.

(This widget is also where you can expunge all of those wireless networks that you connected to once in order to see what they were, and now you never want to touch them ever again and certainly don't want your machine to try to associate with them every time it sees them. How handy it is that nm-applet puts it in such an obvious place as configuring VPNs, gives you no other access to it, and has no help that you can see if you are, perhaps, not connected to the Internet. It's almost as handy as system-config-network, which is an achievement that Gnome should be proud of.)

Now you need to set some DSL PPPoE parameters.

In Toronto at least, DSL identifiers are in the form '<id>@<something>', where the second part specifies your ISP; I believe that I've seen this called the (DSL or authentication) realm that your ID is in, and I believe it's conventional for the realm to look like a domain name, eg bell.ca. In the nm-applet widget, this all goes in the Username field, the Service field is usually blank, and you put your password in the Password field. As usual, tick the 'Show password' box so that you can actually make sure you're entering the password right. Finally, pick a useful name to put in the Connection name field at the top.

(I also set the 'Send PPP echo packets' option in the PPP tab, and it's possible that this is important.)

If you're doing this with the machine completely disconnected, don't be surprised when your new DSL connection doesn't show up in the network manager applet's list of available networks. It's smart enough to know that a PPPoE DSL connection depends on having a connected physical network and not list them if there isn't one.

You can have multiple PPPoE DSL connections defined (with different names).

The network manager applet stores almost all of this information in gconf under the key /system/networking/connections/<N>, where <N> is some number; you can directly edit or remove things in the traditional manner. I believe that passwords are stored in the depths of the Gnome keyring system.

I am not sure I know all of the places where system-config-network stores information, but one of them is in files under /etc/sysconfig/networking (especially devices/); you can use this to remove entries that you unwarily added through it (you may also need to remove things from the CHAP and PAP secrets files in /etc/ppp). You might think that you could delete entries from inside the program, but apparently not, at least for me; once entered, entries cannot be deleted or, for that matter, activated. Instead they stick around like glue to contaminate various other things until you get rid of them by hand.

(I'm writing this down now because if I didn't and I ever needed it again, I would hate myself very much.)

Sidebar: Toronto's special test DSL realm

At least in Toronto, there is a special test DSL user and realm that exists on every DSL service point; it is user 'test@test' with password 'test'. If your local system and the DSL service point are working properly, establishing a PPPoE DSL connection with this identity will give you an (unroutable) IP address; what local and remote addresses you see depends on what DSL service point you're using, and Bell techs can use this information to identify what exact bit of Bell equipment you're being connected through.

(I don't know if this special test user is handled by your DSLAM or by something somewhat upstream of it.)

This test user makes a useful check to see what's broken during local troubleshooting. If you can get a connection as test@test but your regular DSL connection doesn't work, it's almost certainly your ISP's problem (or a problem between Bell and your ISP). If you can't even get connected as test@test, the problem is closer to home; local software, bad DSL signal, or things gone sufficiently wrong inside Bell's systems.

(You can also use it for basic performance testing for things like packet loss and packet timings, and for pinpointing the likely source of connection stability problems.)

Of course none of this is very important if you're using Bell as your DSL ISP, but why on earth would you want to do that? If you're reading this (and are in Toronto), there are much better choices.

The timing of production Linux deployments

Yesterday I wrote that even in two or three years, building on btrfs would be risky because we'd probably be one of the early people doing serious production deployments using it. I want to note something related to that statement: what I said has almost nothing to do with how ready or not btrfs will be in two or three years, and everything to do with timing and release schedules.

Production deployments are almost invariably based on Linux distributions with long term support, the most prominent being Ubuntu LTS and Red Hat Enterprise Linux (or CentOS). These distributions have quite long release schedules, so even if btrfs magically got perfect tomorrow, it would likely be years before it would ship as part of such a distribution, much less proven in production.

Well, sure. But how many years?

Per here, Ubuntu does LTS releases every two years. The most recent LTS was released in April 2010, so the next Ubuntu LTS will be released in spring 2012. Red Hat doesn't publicize any release schedule for RHEL, but per the timeline here releases have been running two to three years apart. RHEL 6 was just released at the end of 2010, so the earliest I'd expect RHEL 7 is at the end of 2012 and the end of 2013 seems more likely to me.

Then there are two complications. The first one is that 'available in a distribution' is not the same thing as 'proven in production'. To go from one to the other takes time; it takes a year or two for people to deploy something based on a new distribution and have positive long term experiences with btrfs. So even if a fully ready btrfs was available in 2012, deploying it in 2013 could easily make us one of the relatively early adopters in production deployments.

(Deploying in 2014 would be safer, but I'm not sure our fileserver infrastructure will last that long; the fileserver hardware would be approaching six years old at that point.)

The second complication is that long term stable releases do not use the latest and greatest software versions as of when they're released. Instead they use older versions (often significantly older), because they freeze things months before release in order to have enough testing and debugging time. Similarly, the distributions have to make decisions on what to support and emphasize months before they release; no one is going to add btrfs as a first class, fully supported filesystem at the last moment. So what matters is not btrfs's state right at release, but its state significantly before release.

Pragmatically, I'd expect that btrfs would have to be stress-tested in at least one Fedora or regular Ubuntu release before it would appear as a fully supported feature in RHEL or Ubuntu LTS. This potentially puts another delay in the process, since the Fedora and Ubuntu release managers are not going to do this until they feel that btrfs is pretty ready for the job. This would add another six months or so of delay, since both Ubuntu and Fedora on a roughly every six month release schedule.

(How much actual delay it adds depends on when btrfs becomes production ready relative to the release schedules of RHEL and Ubuntu LTS. If btrfs was magically production ready tomorrow, there would be plenty of time for it to be tested in Fedora and Ubuntu releases before the next version of RHEL and Ubuntu LTS were frozen.)

The corollary of this delay is that if you're seriously considering using btrfs in the near future because you need it, you should be keeping an eye on it now and planning some testing as soon as it looks good enough. Waiting until it ships as part of RHEL or Ubuntu LTS will only cost you time.

(This feels a bit odd for old-time sysadmins. It used to be that the big vendor releases had the newest and best stuff; now in Linux they're months behind the times.)

A fun Linux kernel pty problem, or maybe it's not a problem

Every so often I run into a Linux kernel 'bug' where I am not sure if it actually is a bug, or at least where I am not sure if the Linux kernel people would agree with me about it being a bug. Recently, one came up with the pty handling code in recent kernel versions (I don't know how far back it goes, but it's present in 2.6.32 in at least some configurations).

I have a sort of terminal emulator program that wants to allow you to edit the text of the line you're entering 'locally', inside the terminal emulator, instead of having to rely on your shell or program to provide line editing. As part of this, it needs to write your final line of input to the pty without having it echoed (because it has already made it visible to you as you were writing it); however, it wants to leave the underlying pty terminal mode looking as normal as possible for as long as possible, in case a program that you're running cares.

If the program didn't care about keeping pty modes normal, it could just turn off ECHO on the pty. Since it does care, it leaves ECHO on normally; when you tell it you're done and the input line should go to your shell or whatever else is running on the pty, it turns echo off on the slave side of the pty, writes the input line to the master side of the pty, and then turns echo back on on the slave side of the pty.

On old versions of the Linux kernel and on other Unixes, this works and it acts as if the sequence of operations are packetized (across both the master and slave sides of the pty); there is a 'turn echo off' packet, a 'here is some data' packet, and a 'turn echo on' packet, and they are all processed in order. Since the data packet is completely processed before the 'turn echo on', nothing from it is echoed. On current versions of the Linux kernel, some or all of the input line is echoed; how much is echoed depends on timing. The kernel is now acting as if 'turn echo off' and 'turn echo on' are instant operations that take effect the moment you make the system call; they are not interleaved with data operations. If there is unprocessed input data when echo is turned on, it gets echoed.

(You can find a test program for this here.)

On the one hand, this is a behavior change from prior Linux versions and from what other Unix versions do. On the other hand, it sort of feels as if this has always worked by accident since the more I look at it, the more peculiar it feels. So the net result is that I am not sure that the new behavior is actually a bug, instead of just a change that stops doing something odd and wacky that previously worked more or less by accident.