Wandering Thoughts archives

More on Linux FHS and /var

Through my Referer logs, I stumbled over this reply to my earlier entry on the FHS, which points out /srv as the answer to my criticisms. However, as the author of it notes:

[...] Then again I haven't seen any package management that does something to /srv [...]

Well, yeah. That's the problem; no one uses it. The FHS in theory might include /srv to deal with my exact criticisms, but it doesn't in practice because no one uses it. This is more than a matter of semantics and a little bit of configuration annoyance; as a commentator on my SELinux entry noted sort of indirectly, people have built software that assumes you are following the in-real-life FHS and thus are putting your data in the duly designated places in /var.

This points out my ultimate problem with the FHS and people assuming that it is always right: these people build software that requires you to follow the FHS without pausing to find out if people actually follow the FHS in real life and if the decisions that the FHS mandates actually make sense. When I don't follow the FHS for various good reasons, this software blows up in my face and makes me grumpy until I turn it off.

(I'm sure that there's people following the FHS and putting real data in /var; it's the path of least resistance, machines these days have huge disks anyways, and if you don't have that much data and you don't have significant infrastructure, you don't really care about these issues. I'll admit that we have some machines set up that way for exactly that reason.)

Update: I've been both unclear and somewhat too extreme here. See the comments for details.

Linux's FHS is not the right answer for where to put data directories

I think that the FHS is a good attempt, and any standard like that certainly has to say something about where programs like databases and web servers should put their data files by default. But at the same time it is stupid, and you do not want to follow it in serious production; instead you really do want to relocate the data directories of various programs to different places, even in the face of things like SELinux.

(Of course I argue that the solution is to turn SELinux off.)

For a start, the FHS puts everything in /var. This is sort of forced on the FHS by its desire to not invent lots of new directory hierarchies and the general role assigned to /var, but this doesn't make it a good idea. If you are doing anything significant, you really want to put your various pieces of data into their own filesystems so that you can size and manage them separately (and so that /var filling up doesn't take out your database, and your database filesystem filling up doesn't take out your system logging).

(If you're worried about having to statically partition your systems, just use LVM and leave some free space in your LVM logical volumes.)

Once you're using separate filesystems, you don't want to nest them inside /var; it's harder to manage, more fragile, and generally more annoying. Putting your new filesystems closer to the root filesystem, generally in some local directory structure of your choice, is a lot easier in the long run. This is definitely the case if you think that you will ever want to NFS-mount any of these filesystems.

(Also, increasingly people are building systems that don't cope well with filesystem mount ordering dependencies more complicated than 'mount the root filesystem first'.)

The final reason to avoid /var is precisely because it is the FHS standard. To summarize what could be a long entry, if you care about stability and change management you want to be in exclusive control of your data directories; you don't want to be sharing them with anything else, including the native package system. Standard data directories in /var/ are inevitably shared between what you're doing to them and what the package system is doing to them.

(Yes, perhaps your package system is not supposed to touch files in /var after you've started changing them. Quick, is it smart enough to not restore files that you've deliberately removed? Do you have to remember to do some magic dance when you remove a file just to be sure that it won't come back again in a package update? You can likely make the standard /var directory work, but using your own directory is much simpler and more resilient against the inevitable oversights.)

None of this applies if you're not going to change the contents of the /var directory yourself (either by directly editing things or by working through the program). Unless they're large enough to cause problems or need special performance, you might as well leave things like nameserver caches, mailer spool areas, and so on in /var.

One of SELinux's problems is that it's a backup mechanism

It has recently struck me that one of SELinux's less obvious problems is that it is a secondary, backup security mechanism. By this I mean that no current system depends on SELinux for its primary security mechanism. Instead the primary security mechanism for Linux is still Unix permissions and UIDs, and on standard distribution setups SELinux is there purely to limit the damage if a program has a bug.

(One of the signs of this is that SELinux denials are treated as a big deal. Imagine if your Linux system alerted you every time a program got an EACCES when it tried to open a file.)

One of the consequences of this is that SELinux suffers from a serious false positive problem with its alerts (for more background on this in general, see here). Because exploits against vulnerable programs are such a rare occurrence, almost all SELinux alerts are in fact either bugs in your distribution's setup of SELinux or because SELinux is fragile in the face of system changes. This both annoys people and conditions them to assume that SELinux itself is the root cause of any problem that it reports.

Another consequence is that time spent dealing with SELinux is almost always wasted time. Since it is only a backup mechanism against what is in practice a very rare event, SELinux spends most of your system's life doing nothing useful. Time spent maintaining something that does nothing useful is wasted time, a deadweight loss. To the extent that you don't have to do anything to maintain SELinux, it is cheap insurance; to the extent that you do have to do things to maintain it (and may have to learn about an entire complicated system), it is expensive insurance that can only be justified if you have an unusual risk profile. Turning SELinux off if it causes you problems is, I'd argue, a completely rational response to the situation unless you can maintain it very cheaply.

(Remember, staff time is not free. Often it is very expensive in practice, because it is finite and not expandable.)

As a secondary, backup system SELinux is pretty much always going to be somewhat neglected. People involved in Linux, from sysadmins to software authors to distribution packagers, don't have to deal with it the way they have to deal with Unix permissions (and most of them should not have to, not unless it becomes Linux's primary security mechanism, which is pretty unlikely to put it one way). This neglect means that SELinux is always likely to have problems, problems that almost never exist with Unix permissions.

(If someone makes a mistake so that Unix permissions are wrong, it gets noticed right away. SELinux, not so much.)

All of this compounds on itself to some degree. Distributions are trying to push this particular rock uphill by turning on SELinux's enforcing mode by default; if it works, this is virtuous circle where widespread enforcing mode makes people get the SELinux configuration right, which makes it less work to run SELinux in enforcing mode, and so on. But it can go the other way too, where enforcing mode is too strict so people don't use it, having warnings enabled produces too many because without strict mode fixing the configuration isn't that important, and then people turn SELinux off entirely.

(My personal opinion is that SELinux has one foot in each camp right now. Distributions seem to have mostly succeeded in making SELinux work for simple default configurations and are in the virtuous circle, but non-default configurations go over to the other way almost immediately.)

My problem with SELinux

I think that my core problem with SELinux is that it is very fragile. By this I don't mean that it's buggy and so breaks in normal operation (because as far as I can tell it generally doesn't), but instead that a SELinux configured system is not resilient to changes. If you run the system exactly as your distribution delivered it, with everything in the same places and with the same user-ids and so on, the result generally works.

(You start running into potential problems if you run less common software, because all of its configuration is less well tested.)

However, when you start changing your system all bets are off. Do you want Apache's data to be somewhere other than the system standard? Do you want MySQL to store databases somewhere besides /var, because you have a really big /db filesystem? You generally lose. Unless you know enough to carefully modify your SELinux configuration (and keep the modifications intact when you do things), SELinux will break things; it will stop daemons from starting, it will prevent programs from reading files, and so on. Many of these failures are mysterious ones, where operations that should work fail inexplicably. Often they work if you run them by hand, which vastly complicates troubleshooting.

This is why I say that SELinux is fragile and it makes your system fragile. If you change things, you run a high risk of breaking something. If you are not familiar with SELinux and how your distribution uses it, it is hard to find what is broken and fix it again; it's generally simpler to turn off SELinux entirely.

Part of the problem is that SELinux labeling for files is often partially redundant information; for example, you specify the MySQL database directory in a configuration file, and then you 'specify' it again by putting the right SELinux label on it. SELinux would be significantly easier to work with and less fragile if you did not have this redundancy, so that some part of the SELinux system read your MySQL configuration and automatically set the labels appropriately.

(Then you could change your configuration once, in the way that you expect to, instead of twice. As a sysadmin I don't like redundancy in configuration settings because it's an opportunity for errors and inconsistencies to creep in.)

The optimistic view of SELinux's real purpose

In an entry, Zed Shaw recently wrote:

P.S. I have a long bet that SELinux is an NSA backdoor. Any takers?

I'm an optimist, so I think that there's another good explanation for SELinux and all of the effort that the NSA has poured into it. My theory goes like this:

Basically, the government has a problem. Historically and for good reasons, it is the only customer for really secure systems (ie, rainbow book level paranoid security). This has always translated to high prices, and lately it has translated to more or less vendor indifference to developing that sort of system; not only was the government getting high prices, it was getting out of date systems (if it was getting anything at all).

Given that it was in many ways the future of commercial Unix, Linux presented the government with an even bigger problem than before (Linux raised vendor indifference to government security desires to new and dizzying heights) but also a great opportunity to do what other people have been doing and hitch a ride on general Linux development to lower the costs of developing a secure Unix. All that the NSA had to do was develop a 'colour book' security addon for Linux and then get it accepted into the kernel. Hence, SELinux; while it cost the NSA a bunch of money, it also stands to reduce the government's costs of acquiring secure Unixes in the future (since they can now be built on top of Linux at relatively low cost).

(As basically the only customer for colour-book secure Unix, the government pays its development costs one way or another. Normally the government pays for it indirectly; with the NSA's work on SELinux, it was paying for it directly instead, but hopefully getting more for its money.)

I also suspect that some people inside the NSA had a beautiful dream where the availability of a free colour-book secure system caused people in the outside world to finally understand what it was good for and how much it could help them. Like many dreams of mathematic security, I don't expect this one to get very far (partly for the reasons covered in the last entry).

PS: it continues to amaze me that so many people run with SELinux turned on. All I can think is that they must be happy to run their systems with everything in the stock locations, or they have a lot more time and interest in learning how to control SELinux than I do.

(I believe I still have SELinux turned on on my Fedora laptop as an experiment. But the laptop runs a basically completely stock desktop environment, with no databases or daemons or whatnot. And even then I periodically get SELinux warnings.)

Sidebar: why secure Unixes will still cost the government money

The government doesn't give coloured-book security ratings to software, only to systems. SELinux and Linux are software, so someone still has to package and assemble them into a specific system, with specific configuration and setup and so on. That someone is going to want money for doing this (and for going through the hassle and expense of getting the result certified).

This system certification process is one of the reasons that government colour-book secure systems were always rather behind the times; they were basically specified and put together years before they can actually be used. If you are really cautious and care a lot more about non-disclosure than availability, this is an acceptable tradeoff.

On improved but less functional versions of things

Every so often I wind up reading something that makes me see red. Today's is this piece (seen via Hacker News). So I want to write an open message:

Dear open source people, it is really simple. If someone has a working system at time X, upgrades to your new software, and their system stops working, they do not care if their system might work at some undefined point in the future. You broke their system now, and this is what they care about. You cannot win their hearts back by telling them that everything will be even better than it was before at some point in the future. Instead, the inevitable result of breaking things now is to cause people to abandon your software and you, because most people want a working system more than anything else.

(People do not even care about exactly why their system broke, no matter that open source people think that they should, but this is a separate rant.)

This doesn't mean that you can't release your new stuff before it is completely baked and ready to replace the old stuff; release all you want, just with appropriate cautions. It does mean that you should not start advocating replacing the old stuff with your new stuff (no matter how convenient it would be for forcefully recruiting unwilling beta testers) until it really is a fully functional genuine upgrade. Ideally you should think about how to run both the old and the new systems in parallel, so that you can easily recruit testers and so that distributions will have an easier time of making your new, unfinished stuff an option.

In the Linux world, software authors don't shoulder the major blame for this 'let's throw the users to the wolves' upgrade enthusiasm. Distributions play a significant part, because it's distributions that usually make the final decisions. (Sometimes their hands are cleverly forced by software authors who take advantage of 'must have' upgrades to slipstream in new systems as dependencies.)

The poster child for this entire unpleasant phenomenon is Linux audio. I believe that I have gone through no less than four different audio cards over the course of two years or so, as upgrades to kernels or PulseAudio or both broke my current setup and forced me to find some other piece of hardware that today's software likes. I cannot say that this experience has left me very happy, and I am almost out of audio cards.

PS: this is not unique to open source. Very few vendors can get away with going from working to broken and then telling people to suck it up for some amount of time (especially if the amount of time is unknown). When vendors can get away with it, it is pretty much because their users have no options and could not abandon the vendor under any circumstances. Users placed in this situation tend to devote a lot of energy to getting options, at which point the vendor can become very unpleasantly surprised. (Being hated by your customers is not a good thing.)