Wandering Thoughts archives

What Linux distributions we use (and sort of why)

A friend is currently polling people about what Linux distributions they use and why. Since my answer is moderately long and I dislike editing in browsers and I actually want to capture this for future use, I'm putting it here.

On servers, I have direct experience with Red Hat Enterprise Linux, Fedora Core, and Debian. We use RHEL instead of CentOS mostly because the university arranged a site license, so it's effectively free to use. RHEL gets used on a lot of mission critical bits of infrastructure, including the IMAP server pool for the campus wide email system.

On workstations (depending on how you define that), I have Fedora Core, Debian, and a recent Ubuntu test install in a VMWare image. We've used FC in student labs as well as for sysadmin machines, and I use FC on my home machine.

My ideal Linux distribution would have:

1. relatively frequent releases with relatively current software (at least one release a year)
2. three or four years of security fixes for old releases
3. easy upgrades from one release to the next.

Debian misses out on #1 in a big way, which has caused me a bunch of pain in the past.

Fedora Core does #1 and #3 but skips #2 (although there's always the Fedora Legacy project). This means I like it on workstations and relatively constantly evolving servers, but I don't want to bother with it on relatively static servers.

RHEL makes me wary about #1 and a bit of #3 but nails #2 to the wall, making me like it a lot for servers I want to throw in a corner and forget about.

Ubuntu looks promising, since they claim to be doing all three. I'm a little bit wary of trusting this; after all, back around the Red Hat 7.x time, Red Hat looked like it would do all three too. And doing all three is a lot of work.

Sidebar: RHEL versus CentOS

I consider CentOS and RHEL equivalent since they are built from the same source base and the CentOS goal is more or less to have exactly the same binary bits. RHEL may have a better integrated and slightly more prompt update system; on the other hand, I would sort of rather use yum anyways.

The one big thing you get with RHEL that you don't get with CentOS is various forms of official paid for support and certification. If you need an officially signed off 'guaranteed to work, supported by the vendor' Oracle installation, you can't get by with CentOS.

SELinux bites man: a story

A co-worker recently came to me with an interesting and mysterious problem. He was setting up a Red Hat Enterprise 4 based machine with MySQL, using a default setup except he'd changed the location of MySQL's data directory from /var to a different partition. Now MySQL wasn't starting properly.

The symptoms were really funny: the init.d script installed by the system didn't work, except if he ran it by hand with 'sh -x', it did. So he renamed it out of the way and grabbed a copy from another RHEL4 machine with a working MySQL, which worked. But it turned out that the two scripts were identical. So why did the new one from the other machine work but the old one from the install not work?

(This was where he called me in.)

Fortunately I had been recently reading a series of articles on SELinux; something about the whole situation tickled the back of my mind, and a little light labeled 'file contexts' lit up. A quick lsattr showed that the two scripts had different contexts; the 'as installed' one had a special context, and the one copied from the other system had a generic one.

And this was the problem: the SELinux MySQL context lacked the magic SELinux permissions to access the new data directory location, because (of course) the new location hadn't been SELinux labeled as a MySQL area. However, the normal root context could access everything fine.

(In Red Hat's SELinux setup, many daemons are deliberately run with extra SELinux-imposed restrictions so that if someone finds and exploits a vulnerability in the daemon it does less damage.)

So when the original MySQL init.d script was run directly, it switched into the MySQL SELinux context and failed to access its data directory. However, special contexts on a shell script only get switched into when you execute the shell script directly, so when the original script was run via 'sh -x' it was instead running in the normal root context, could access its data directory, and worked fine. The copied script always ran in the normal root context since it was not specially labeled, so it worked fine all the time.

(We tested this guess by running the original init.d script with just plain 'sh mysql start' instead of 'sh -x mysql start', and it worked fine then too.)

I believe my co-worker's workaround was to turn off SELinux, on the grounds that he didn't want to try wrestling with that particular pig right then.

Pointers to some SELinux explanations

SELinux is one of those things that have been cropping up on my radar ever since I had to start telling the Red Hat installer not to turn it on. (I kept not enabling it because changing an entire security architecture of a system is not to be done lightly, even when it's a new system I'm setting up.)

SELinux is an imposing system with equally imposing documentation. Fortunately, recently Dan Walsh of Red Hat has been posting some very useful (to me) 'SELinux for beginners' documentation:

1. a basic introduction
2. How does SELinux enforce policy?
3. Applications that work with SELinux
4. File contexts and mv/cp/install
5. How logging in and so on work
6. The /etc/selinux/config file and how to change what SELinux level your system uses
7. config files in general
8. Managing file contexts
9. the restorcond daemon
10. SELinux manpages and AVC messages
11. Booleans
12. Role Based Access Controls (RBAC)
13. Using RBAC in a MLS policy
14. more on Module handling
15. SELinux reveals bugs in other code
16. Loadable Modules - File Context

There's also a Fedora Core 5 SELinux FAQ, with links to other FAQs. However, from reading through it I think Dan Walsh's stuff is easier to follow.

(This entry is a bit belated, because Dan Walsh didn't so much wrap up his series of entries as stop writing them, which I can't exactly blame him for, and I was sitting on it until the series was 'complete'.)

Why I like 'lib64' better than 'lib32' for the x86_64 transition

One of the small controversies in Linux on the 'x86_64' architecture is where 'native' 64-bit libraries should go, versus where 'legacy' x86 libraries should wind up. One version is what Fedora Core does, putting 64-bit libraries in /usr/lib64; the other version is what Debian does, where legacy x86 libraries go in /usr/lib32 and /usr/lib is for 64-bit binaries.

(Some distributions use more complicated paths than /usr/lib32.)

I used to be pretty neutral on this whole issue, but then I got an AMD64 machine and started playing around in the whole bi-arch world. Now I think the 'lib64' approach is the right way.

What it comes down to for me is that x86_64 is not really a new architecture; it is effectively an upgrade of the existing x86 architecture. (It being an upgrade instead of a wholesale replacement is why it's interested to start with; I have little desire to jettison all of my x86 programs in a great big lurch. Been there, done that, it's a pain.)

The 'lib64' approach is the right one for an upgrade, because it leaves the old stuff alone and puts the new stuff in a new place. It's even possible to (re)use x86 RPM packages directly, since they install in the same place (and work the same) regardless of whether they're running on an x86 OS or an x86_64 one. This can't be done on a 'lib32' system; you need to rebuild x86 packages for x86_64 just to make them put libraries in /usr/lib32 instead of /usr/lib.

(I suppose you could put a bunch of magic into your packaging system, but this soon starts to get ugly.)