Wandering Thoughts archives

SELinux bites man: a story

A co-worker recently came to me with an interesting and mysterious problem. He was setting up a Red Hat Enterprise 4 based machine with MySQL, using a default setup except he'd changed the location of MySQL's data directory from /var to a different partition. Now MySQL wasn't starting properly.

The symptoms were really funny: the init.d script installed by the system didn't work, except if he ran it by hand with 'sh -x', it did. So he renamed it out of the way and grabbed a copy from another RHEL4 machine with a working MySQL, which worked. But it turned out that the two scripts were identical. So why did the new one from the other machine work but the old one from the install not work?

(This was where he called me in.)

Fortunately I had been recently reading a series of articles on SELinux; something about the whole situation tickled the back of my mind, and a little light labeled 'file contexts' lit up. A quick lsattr showed that the two scripts had different contexts; the 'as installed' one had a special context, and the one copied from the other system had a generic one.

And this was the problem: the SELinux MySQL context lacked the magic SELinux permissions to access the new data directory location, because (of course) the new location hadn't been SELinux labeled as a MySQL area. However, the normal root context could access everything fine.

(In Red Hat's SELinux setup, many daemons are deliberately run with extra SELinux-imposed restrictions so that if someone finds and exploits a vulnerability in the daemon it does less damage.)

So when the original MySQL init.d script was run directly, it switched into the MySQL SELinux context and failed to access its data directory. However, special contexts on a shell script only get switched into when you execute the shell script directly, so when the original script was run via 'sh -x' it was instead running in the normal root context, could access its data directory, and worked fine. The copied script always ran in the normal root context since it was not specially labeled, so it worked fine all the time.

(We tested this guess by running the original init.d script with just plain 'sh mysql start' instead of 'sh -x mysql start', and it worked fine then too.)

I believe my co-worker's workaround was to turn off SELinux, on the grounds that he didn't want to try wrestling with that particular pig right then.

Pointers to some SELinux explanations

SELinux is one of those things that have been cropping up on my radar ever since I had to start telling the Red Hat installer not to turn it on. (I kept not enabling it because changing an entire security architecture of a system is not to be done lightly, even when it's a new system I'm setting up.)

SELinux is an imposing system with equally imposing documentation. Fortunately, recently Dan Walsh of Red Hat has been posting some very useful (to me) 'SELinux for beginners' documentation:

1. a basic introduction
2. How does SELinux enforce policy?
3. Applications that work with SELinux
4. File contexts and mv/cp/install
5. How logging in and so on work
6. The /etc/selinux/config file and how to change what SELinux level your system uses
7. config files in general
8. Managing file contexts
9. the restorcond daemon
10. SELinux manpages and AVC messages
11. Booleans
12. Role Based Access Controls (RBAC)
13. Using RBAC in a MLS policy
14. more on Module handling
15. SELinux reveals bugs in other code
16. Loadable Modules - File Context

There's also a Fedora Core 5 SELinux FAQ, with links to other FAQs. However, from reading through it I think Dan Walsh's stuff is easier to follow.

(This entry is a bit belated, because Dan Walsh didn't so much wrap up his series of entries as stop writing them, which I can't exactly blame him for, and I was sitting on it until the series was 'complete'.)