Wandering Thoughts archives

The risks of using CentOS are split

Recently (for my value of recently) I wound up reading Matt Simmons' entry, CentOS 6 - Great, but for how long?, in which he worries about the delay in CentOS releases compared to RHEL, especially the CentOS 6 release, and what happens if CentOS updates stop being timely. My reaction is that there are two very different risks being conflated here, because major releases like CentOS 6 are not at all like updates to existing releases.

CentOS is strongly based on RHEL, to the point where it tries for binary compatibility, and Red Hat publishes source RPMs for RHEL and RHEL updates (see here). This makes updates for existing CentOS releases a relatively low risk thing; if CentOS flakes out, in theory you can simply take the RHEL source RPMs for the updates, rebuild them, and install them (you may need to update some explicitly declared dependencies if there are slightly different package release numbers). As far as I know, RHEL point releases are simply a rolled up set of current package updates, so building your own CentOS point release and updating machines to it is equally easy.

(We certainly have 'RHEL 5.x' machines that were installed as RHEL 5.0 machines and have been updated ever since then. It is possible that they are slightly different than what we'd get if we reinstalled them from a RHEL 5.x ISO image, but if this is the case I actually consider it a bug in the RHEL 5.x ISO image and I would be just as happy to avoid it.)

The CentOS people themselves have to do somewhat more work than this, but that's because they need to publicly distribute the result of their work and so they need to strip out Red Hat's trademarks and other branding and add their own. If you just use your rebuild RHEL RPMs internally, you're unlikely to care about this issue and so you don't need to do this time consuming work.

(You will have to keep track of Red Hat security updates yourself and so on, though.)

Major version updates (such as creating CentOS 6) are the slow, risky thing. Because they are major version updates you can't just rebuild the RPMs in an existing environment to get the same results if CentOS is too slow, and bootstrapping an entire Linux distribution is somewhat tricky (especially if you're trying to be binary identical to another one). However, these major version 'updates' are exactly the point where it's easiest to migrate to a different base distribution, because as far as I know Red Hat still doesn't support major version to major version updates (and thus neither does CentOS); the official way to move from RHEL 5 to RHEL 6 (or CentOS 5 to CentOS 6) is to reinstall your machines, and if you're reinstalling anyways you can (re)install anything that you want to.

(By the way, this bootstrapping difficulty is likely to be one major reason that CentOS is slow with major version updates.)

PS: from my perspective, the important delay to track for a RHEL derived distribution is the delay between Red Hat's release of a security update and when the derived distribution releases its own updated binary RPM. This is the delay that you really care about if you're running the derived distribution. I am not sufficiently energetic and interested to try to generate these numbers for, eg, CentOS or Scientific Linux.

(I care mostly about security updates because while plain bugs matter, they are generally far less a problem if your distribution is slow. And the delay in releasing the RPMs that brand your machine as being a new point release (eg, 5.x for some new x) are the least important updates of all.)

(To answer a potential question: per WhatLinuxDistributions, we use RHEL here because we have a campus-wide site license for it. As a result I have not looked in detail at any RHEL derived distribution.)

Ubuntu, illustrating how to utterly fail at kernel security updates

Just like last time, this is a rant.

Here's a simple procedure that you too can follow:

• develop fixes for some CVE issues in the various kernel versions that you maintain.
• announce and release new kernel security updates for most of your currently supported distributions, but not for 10.04 LTS. Trained sysadmins running 10.04 will ignore them, and even if they don't there's no 10.04 updates for them to apply.
• announce and release new kernel security updates for specialized versions of 10.04, like the one that runs on Amazon's EC2. Trained sysadmins will ignore them and even if they don't, there's still no general 10.04 update.

• realize that oops, you forgot to do a general 10.04 kernel update.
• quietly put a kernel update into your security repository. Do not announce it on your security mailing list, because that might be embarrassing. Besides, who reads those things anyways? People using Ubuntu should just install every available update, and right away; as we've already established, putting useful details in kernel security announcements is optional anyways.

(Yes, I checked the ubuntu-security-announce archives just to make sure that our mail system hadn't swallowed an announcement.)

Of course, as a standard thing it appears that the Ubuntu changelog for the kernel package doesn't necessarily include a list of the CVEs that are fixed in any particular revision of the package. It would make sense that the specialized versions have the same bugs fixed as the main one, but given past issues it's hard to tell unless I spend far more time on this than I have any interest in doing.

Right now, I am very, very angry with Ubuntu's slipshod practices. We run Ubuntu on servers, in an environment where we can't just reboot machines because Ubuntu feels that we should; we really do need to be able to assess the importance of security updates, especially when many of the bugs don't even apply to our specific environment (they require protocols we don't use, hardware we don't have, or the ability to plug devices with maliciously corrupted filesystems into servers in a machine room).

PS: my story may not be exactly what happened (see eg), but from the outside it sure looks like a plausible story.

PPS: there was also a stealth 8.04 kernel security update, but it's less clear what's going on with that one. On one hand, the package changelog between 2.6.24-29.90 and 2.6.24-29.91 lists a whole bunch of CVEs. On the other hand, many of them are old. Did Ubuntu miss a significant set of security updates for the 8.04 kernel, and only notice things now? Ubuntu 10.04 had fixes for at least some of these CVES back in March.

(See also the tracking bug for the 8.04 update.)

Update: Ubuntu has now released two notices for this kernel update, for the 10.04 version and for the 8.04 version. It remains the case that Ubuntu published security updates in the repository well before an announcement was made (apparently surprising some people), and that they let a general kernel package update lag well behind at least specialized version updates.

Why I'm going to be skipping Fedora 15

I've pretty much decided that I will not be upgrading my machines to Fedora 15 or installing it on new machines; they will stay at Fedora 14. The short answer to why is 'Gnome 3', but not quite for the reason that you might think.

I wrote earlier about how Fedora 15 has a package dependency failure in the sshmenu applet that's an important part of my nice ssh environment for Gnome, ultimately due to the API changes from Gnome 2 to Gnome 3. At the time I casually said that it shouldn't be too much work to write a Python version of the sshmenu applet, since Python was still supported in Gnome 3.

(Given how much of Fedora's various GUI tools are written in Python, I'm pretty sure that Fedora 15 would not have Gnome 3 unless the Python bindings were in good shape.)

As it turns out, I'm both right and wrong. I'm right in that it's not too much work to write a Python equivalent of the Ruby sshmenu applet (although it was kind of annoying, much like all GUI programming seems to be). I'm wrong in that Gnome 3 doesn't support Gnome 2 style 'applets' at all. I didn't notice this because of course I developed my applet on my regular desktop, which is still running Fedora 14; I only discovered this when I got my applet into a decent enough shape that it was worth trying it on a Fedora 15 virtual machine, at which point I discovered that the system was completely ignoring its service definition file.

The Gnome 3 shell environment doesn't support applets because it has a completely different approach to applet-style panel additions. All of the old bonobo-based applet APIs are completely gone, as is support for writing applets in the language of your choice (assuming it has panel API bindings). In Gnome 3, the equivalent is 'shell extensions', which are written in Javascript (with some degree of CSS for UI chrome). Old applets have to be completely rewritten (and as usual the Gnome 3 developers seem rather dismissive of the more interesting ones, which may have implications for what the shell extension API lets you do).

(This means that it was a red herring that the Ruby bindings haven't been ported to Gnome 3; even if they had been, the applet itself wouldn't work any more.)

This has a number of implications. Obviously, developing a version of the sshmenu applet for Fedora 15 is now a lot more work than I was expecting. Also, I need to be working in a Fedora 15 Gnome environment to develop it, which has certain bootstrapping problems. Finally, this change has removed a lot more than the sshmenu applet; as a start, Fedora 15 also lacks the 'Command Line' applet that I rely on for the other part of my nice ssh environment for Gnome. Getting my usual customized Gnome environment back in Gnome 3 is clearly going to take a bunch of time, if it's even possible.

In theory this only affects my work laptop (the only second sort of machine that I have); my home and work workstations use a completely customized non-Gnome environment (and my home is already far out of date and the hardware needs to be replaced anyways). In practice, I've traditionally used my laptop as the pilot for Fedora upgrades and I'm not happy with any of my options here; I get to choose either a blind Fedora upgrade on my primary machine at work and my laptop being stuck behind my workstation, or losing a significant amount of what makes my laptop convenient.

More generally it just seems that right now is too early for Gnome 3; people just haven't figured out how to duplicate lost functionality, updated language bindings, ported programs to the new APIs, written good documentation and howtos, and so on. Coming back in six months seems much more likely to get me a well developed Gnome 3 ecosystem than trying to use Fedora 15 today.

(Of course it's not clear that the laptop even has enough graphics power to run Gnome 3 in the first place, since Gnome 3 demands good 3D acceleration. And I don't know how well Gnome 3 supports such a 'lacking' environment.)

PS: the Gnome 3 fallback environment for machines without good 3D graphics (such as virtual machines) does not run Gnome 2 panel applets, although the look is still relatively classic Gnome.

Sidebar: links and references

(for my own future use if nothing else)

• a discussion of the Gnome 3 shell versus applets
• some documentation on Gnome shell extensions

• a somewhat disturbing Gnome developer view on 'applets'
• more disturbing things from Gnome developers about shell extensions, from LWN

• the status of Gnome language bindings
• some bits on Python on Gnome 3
• Gnome 3 code examples in various languages

• Gnome 3 tips

mxiostat: second generation better Linux disk IO statistics

A while back I wrote and put up xiostat, a program that I wrote to give us a faithful recounting of the Linux kernel's disk IO stats after we discovered the problems with iostat's numbers. Mxiostat is the second generation of xiostat, in part because it will report on multiple disks at once (hence the 'm', for 'multiple').

Because it is now 2011 instead of 2006, I have put the mxiostat source code into git and published it on github as siebenmann/mxiostat. It's a bit minimal and needs some work (like a manual page), which I may or may not get around to someday. Usage information is in the comments at the top of mxiostat.py; information on what the stats are is in comments at the bottom of mxiostat.py and in DiskIOStats.

(Someday I may have a page for it on here too, but we'll see. It's pretty tempting to just use github for all of that.)

Why Ubuntu's PAM versioning failure matters

As I really should have expected, a number of people showed up in the Ubuntu bug for their PAM versioning failure to say that this was no big deal and other people should relax about it. I disagree.

This PAM ABI incompatibility was not introduced by the upstream package (in a patch or a minor version update); it was introduced by Ubuntu's own patching. Ubuntu makes a big deal out of doing only minimal changes in their LTS releases (so minimal that they generally will not fix bugs, only security issues, as far as I can tell). This means that Ubuntu should be carefully auditing the patches that they use.

(If Ubuntu is not carefully auditing patches but is still beating the 'minimal changes only' drum, they are either being lazy or are extremely short of resources. If it is the latter case, they should admit that Ubuntu and especially Ubuntu LTS receives minimal care and attention because they don't have the manpower to do any better.)

That an ABI change slipped through anyways reveals a process failure of Ubuntu's patch auditing. This failure is (probably) the real root cause issue in the PAM versioning failure, not the cron locked up due to a PAM update or even that PAM had an ABI change and it was not detected.

If Ubuntu has had a process failure in PAM patch auditing, it can have a process failure in patch auditing for other packages (and indeed it seems to have had in some cases). Failures in patch auditing are serious; just how serious is revealed by the Debian SSL debacle. If Ubuntu's patch auditing is flawed (or simply only superficial, or even absent entirely), applying Ubuntu package updates in general (not just PAM updates) may be significantly more dangerous than people thought.

Sadly, it's not encouraging that the only actions Ubuntu people talked about in the Ubuntu bug were surface level issues. And really, that's the final reason that this versioning failure matters; it may have exposed something important and disturbing about how Ubuntu people approach maintaining Ubuntu.

(Yes, I'm taking kind of an extreme case view here; the reality is likely to be less scary. But we don't know, and it really does irritate me that Ubuntu's public reaction was so superficial and that people were so accepting of it.)