SELinux bites man: a story
A co-worker recently came to me with an interesting and mysterious
problem. He was setting up a Red Hat Enterprise 4 based machine with
MySQL, using a default setup except he'd changed the location of MySQL's
data directory from
/var to a different partition. Now MySQL wasn't
The symptoms were really funny: the init.d script installed by the
system didn't work, except if he ran it by hand with '
sh -x', it did.
So he renamed it out of the way and grabbed a copy from another RHEL4
machine with a working MySQL, which worked. But it turned out that
the two scripts were identical. So why did the new one from the other
machine work but the old one from the install not work?
(This was where he called me in.)
Fortunately I had been recently reading a series of articles on
SELinux; something about the whole situation tickled
the back of my mind, and a little light labeled 'file contexts' lit up.
lsattr showed that the two scripts had different contexts; the
'as installed' one had a special context, and the one copied from the
other system had a generic one.
And this was the problem: the SELinux MySQL context lacked the magic SELinux permissions to access the new data directory location, because (of course) the new location hadn't been SELinux labeled as a MySQL area. However, the normal root context could access everything fine.
(In Red Hat's SELinux setup, many daemons are deliberately run with extra SELinux-imposed restrictions so that if someone finds and exploits a vulnerability in the daemon it does less damage.)
So when the original MySQL init.d script was run directly, it switched
into the MySQL SELinux context and failed to access its data directory.
However, special contexts on a shell script only get switched into when
you execute the shell script directly, so when the original script was
run via '
sh -x' it was instead running in the normal root context,
could access its data directory, and worked fine. The copied script
always ran in the normal root context since it was not specially
labeled, so it worked fine all the time.
(We tested this guess by running the original init.d script with just
sh mysql start' instead of '
sh -x mysql start', and it worked
fine then too.)
I believe my co-worker's workaround was to turn off SELinux, on the grounds that he didn't want to try wrestling with that particular pig right then.