SELinux bites man: a story

A co-worker recently came to me with an interesting and mysterious problem. He was setting up a Red Hat Enterprise 4 based machine with MySQL, using a default setup except he'd changed the location of MySQL's data directory from /var to a different partition. Now MySQL wasn't starting properly.

The symptoms were really funny: the init.d script installed by the system didn't work, except if he ran it by hand with 'sh -x', it did. So he renamed it out of the way and grabbed a copy from another RHEL4 machine with a working MySQL, which worked. But it turned out that the two scripts were identical. So why did the new one from the other machine work but the old one from the install not work?

(This was where he called me in.)

Fortunately I had been recently reading a series of articles on SELinux; something about the whole situation tickled the back of my mind, and a little light labeled 'file contexts' lit up. A quick lsattr showed that the two scripts had different contexts; the 'as installed' one had a special context, and the one copied from the other system had a generic one.

And this was the problem: the SELinux MySQL context lacked the magic SELinux permissions to access the new data directory location, because (of course) the new location hadn't been SELinux labeled as a MySQL area. However, the normal root context could access everything fine.

(In Red Hat's SELinux setup, many daemons are deliberately run with extra SELinux-imposed restrictions so that if someone finds and exploits a vulnerability in the daemon it does less damage.)

So when the original MySQL init.d script was run directly, it switched into the MySQL SELinux context and failed to access its data directory. However, special contexts on a shell script only get switched into when you execute the shell script directly, so when the original script was run via 'sh -x' it was instead running in the normal root context, could access its data directory, and worked fine. The copied script always ran in the normal root context since it was not specially labeled, so it worked fine all the time.

(We tested this guess by running the original init.d script with just plain 'sh mysql start' instead of 'sh -x mysql start', and it worked fine then too.)

I believe my co-worker's workaround was to turn off SELinux, on the grounds that he didn't want to try wrestling with that particular pig right then.