SELinux's toxic mistake

June 4, 2013

Okay, it's not SELinux's (the technology's) mistake. It's the mistake of people who support SELinux, what outsiders see as the SELinux community. From an outside perspective it's basically the same thing. The toxic mistake is this:

Seriously, stop disabling SELinux. Learn how to use it before you blindly shut it off.

(via, which is another illustration of this, from @jordansissel.)

Let me translate this: 'the beatings will continue until morale improves'.

When people say 'this security tool is too hard to use, gets in my way, and isn't giving me any real benefits', telling them 'it's great if you only spend more time learning how to deal with it' is doubling down on your problems. If you then compound things by telling people that they are just stupid and lazy, don't be surprised if they immediately tune you out because you're acting like a zealot (you may or may not be one, it doesn't matter to people).

It's been apparent for years that SELinux had serious problems in real life (regardless of what the theory says). For example, it's widely considered standard practice to disable SELinux immediately on server installs (as mentioned in the Twitter thread I got this from). The reason people reject SELinux in its current state is pretty simple: security is not their top priority. Unless you are a high risk target, spending almost any time beating SELinux into shape on your machine is a bad tradeoff and pretty much a waste (partly because SELinux is just a backup).

(If SELinux works when you don't touch anything you might as well leave it on, but turning it off the moment it costs you any time is a perfectly rational reaction. So is turning it off immediately so that it can't waste your time.)

In the face of this, for the SELinux community to feel that people are stupid, lazy, or ignorant for not jumping through the magic SELinux hoops and all would be well in the world if only they would mend their woeful ways is breathtakingly stupid and counterproductive. At a stroke it sabotages almost any chance SELinux might have to actually make meaningful improvements. Not that such improvements would be easy even if the SELinux community listened to what the world was telling them (because it's a hard problem), but if the community did listen they might at least have some sort of chance.

(Not that this sort of blindness is new in security or in general.)

(What I think the SELinux community should be doing is a sufficiently large issue that it doesn't fit in the margins of this entry. Of course it's an open question if SELinux can be saved or is worth saving in general given its origins; I think there's a real argument that SELinux's security model simply does not meet people's real security needs by design.)


Comments on this page:

From 46.144.78.131 at 2013-06-04 03:48:02:

we run most our linux infrastructure (200+ servers and growing) with selinux on except for 4 servers.

This has worked great without special tweaking. We only met a problem with our config tool (cfengine) invoking rsync, so rsync was a client selinux was not expecting at that moment. After some tweaking the problem was cleared.

I stopped having problems with selinux in fedora 11 (and that was in may 2009 according to https://fedoraproject.org/wiki/Releases/11/Schedule?rd=Releases/11).

It works. Really.

From 204.109.63.40 at 2013-06-04 04:58:07:

learn how to use it by youtube, seriously?

From 149.160.192.89 at 2013-06-04 10:30:45:

Unless SELinux works with lustre 1.8 then we can't use it.

https://jira.hpdd.intel.com/browse/LU-857

And as I commented on the original "Stop disabling SELinux" post:

http://youtu.be/bFEoMO0pc7k

From 65.126.15.122 at 2013-06-04 11:39:41:

I would be more than happy to hear actual suggestions in this space. If you want SELinux to be easier then please let me know what you think might help with that. I've been giving talks on it for the last 5 years and I can tell you most people after an hour long talk (or the really in depth 3 hour one) are more than capable of using and understanding the system. The last talk I gave was at the Boulder Area LUG and Boulder DevOPS groups which held a joint meeting for my talk and I received several emails the very next day saying how the information I had given them the night before helped them solve a real world SELinux problem the very next day. Maybe its a problem with educating people, maybe its a problem with complexity of tools. However no one has ever given me a real solution to either problem other than they want the allow this to happen button from Windows UAC. They just throw their hands up and spout on about how its not their problem to make an easy to use system.

By cks at 2013-06-04 12:15:10:

They just throw their hands up and spout on about how its not their problem to make an easy to use system.

Here is the thing: IT'S NOT.

If you want SELinux to succeed, it is your job to make sure that it solves real problems and does so well enough that people accept it (or at least people you care about). If outside people are interested enough or invested enough to help you with this, great. But you can't expect it or count on it. If people are saying 'SELinux makes my life suck', telling them 'tell me how to fix it or shut up' (or some variant like 'please give us constructive feedback') is extremely counterproductive. Your mission, should you want to accept it, is to find out how it makes their life suck and then how you (not them) can fix this.

From 143.48.3.13 at 2013-06-04 13:10:10:

At the very least, I wish the rhetoric would change from "disable SELinux" to "put the service you're having problems with into an SELinux permissive domain." Someone even wrote a custom type for Puppet to handle this for new installs. There's really no good reason to make your ftpd explosive in the event of a remote exploit just because Apache's giving you problems.

From 31.151.153.51 at 2013-06-04 13:33:41:

if selinux got in the way I would be the first one to disable it. It does not, so why would I disable it?

It gives us an extra layer of security. Why would you refuse that on an internet facing server?

From 66.175.95.4 at 2013-06-04 13:55:02:

This is somewhat my go-to example when I talk about SELinux. To me it's counter-intuitive and that's the big rub with SELinux.

Clean installation of CentOS on a database server, up to date with patches yadda yadda. The server has a dedicated set of hard disks for the database content.

You run "yum install mysql-server". Go edit /etc/my.cnf and change the location that you want it to store the data in from /var/lib/mysql over to your new mount point.

You start the service, and it fails without a message to the console. You run mysqld_safe directly as the root user and it works fine. You try it as the service and it again fails.

Nothing appears in /var/log/messages. Nothing appears in /var/log/mysqld.log. Nothing in the root of /var/log at all.

It actually logs it under /var/log/audit/audit.log but you've got to have realised that it's being caused by SELinux to know to look in there for it.

That's the real problem underlying the usability of SELinux for me. It blocks stuff ostensibly 'silently' to the usual sysadmin workflow. With no indication SELinux is getting in the way, it doesn't occur to people to go look under /var/log/audit/audit.log. I've wasted untold hours chasing down weird behaviour because I didn't realise it was something SELinux was blocking, and wouldn't intuitively expect it to.

You've got to remember: SELinux is a niche product. It's Linux only and it's only used on a subset of distributions, ostensibly just the RHES based ones. Many sysadmins are working in disparate environments with a mix of distributions and OSs. Despite that there are a lot of commonalities that the POSIX standards introduce that we get used to. We build up intuition and instincts based on that. To my mind and my experience SELinux behaves contrary to any behaviour by any other application. To make it worse, being on just a subset means application developers frequently just aren't interested in making their software work with SELinux with comments similar to https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPSELinux telling you it's unsupported if you have SELinux on. That means more and more it has to be disabled just to be able to run the software we are required to run on the boxes.

Even when you do twig that SELinux might be getting in the way, AVC is popping up messages that aren't exactly clear: 'SELinux is preventing sshd(sshd_t) "read" var_t'

What that's really indicating is that SELinux is stopping sshd from allowing you to log in using an SSH Key that is stored in authorized_keys. (http://blog.warrenstrange.com/2012/06/getting-bitten-by-selinux-and-sshd.html)

It would be a huge step forward IMO if SELinux started popping up messages on the console when it blocks something, or at worst case pop a message in /var/log/messages.

From 216.105.40.123 at 2013-06-04 18:13:10:

IPtables has the same issue as SELinux. It silently blocks stuff, is complicated, and takes time to learn. Most people turn off the built-in firewall which is enabled by default in CentOS and RHEL.

Yet SELinux gets a lot more complaints somehow.

From 82.197.216.4 at 2013-06-05 07:21:03:

In my view its all about ethics, and as someone who contributes (modest) to the SELinux project i can really relate to Richard Stallmans' fight for free software and free software awareness.

A seatbelt will not prevent a traffic accident. It is annoying, and a hassle to use, but it can mean the difference between life and death.

Internet and shared computer systems are like a highway, you share it with others. If you do not make security a top priority for yourself then do it for others on the highway.

There is nothing more sad to me then to hear about some innocents death in a traffic accident/murder due to negligence by someone else that he shared the road with.

There is nothing more sad then getting your site packeted into a black hole by compromised servers and networks administered by people who do not think security is a top priority for example.

In my view SELinux is not counter intuitive, and it does help solve real security challenges, and it kind of hurts me to see people investing valuable time in making the SELinux experience artificial just because they take arguments like some of the arguments above to heart.

SELinux is good as it is, just invest some time into getting to know it.

..But this article is not about SELinux, it is about ethics..., and i agree: ethics and sense of responsibility is lacking, neglect may seem to pay in the short term but you are risking your, your customers, and others lively hoods.

From 82.197.216.4 at 2013-06-05 11:18:25:

My above contribution to this discussion is incomplete so here is the remainder:

The idea of the site is nice in my opinion, as i do not see anything wrong with raising awareness with people that are genuinely not aware, and are willing to be good citizens of the Internet. (and the focus should be on the low hanging fruit)

The problem is that the site asks you to not disable SELinux for all the wrong reasons. This is not about making someone "weep". In my view this about acknowledging that we are all part of a problem.

Many of us are connected. Connectivity is becoming more important to many of us, and the data we store on connected systems becomes more valuable. To some more important than others, but just because your assets on the network are less valuable than others assets on the network, does that mean you can just go ahead and ruin things for others?

So please, safety first https://thecustodyrecord.files.wordpress.com/2011/10/seatbelt.jpg

From 194.168.32.194 at 2013-06-07 04:27:10:

Comparing SELinux to road safety? You'll be asking us to please think of the children next.

Instances such as moving the MySQL data directory and blocking SSH key authentication are prime examples of why many administrators disable SELinux. Sure, they're isolated cases but they're common enough and doubtless there are plenty of others. When deadlines are involved, most people don't want the additional pressure from chasing down issues like this (particularly when they're in a class that lacks adequate diagnostics), no matter how trivial each one may be to solve individually. Additionally, the net benefit from SELinux appears negligible (since getting exploited due to its absence is a theoretical possibility), whereas the immediate benefit of disabling it is readily apparent (problems go away). Yes, that's short term thinking, but it's also the reality. As Chris says, absent more compelling evidence, I doubt there's much mileage in expecting people to realign their worldview to accommodate your software, no matter how well-intentioned.

From 37.152.227.124 at 2013-06-16 11:23:06:

I'm glad that somebody has stepped up and said this.

SELinux has been a technology with awesome capabilities but limited acceptance for about ten (!) years now. It make systems unmaintainable for anybody that hasn't spent a chunk of time specifically learning SELinux, it looks hard to learn well enough that you can be confident that you can fix it when it breaks your system, and it isn't a high priority enough for lots of people to spend time learning, so good luck getting help.

It's unreasonable for to blame system administrators for not wanting to get into situations like this: http://danwalsh.livejournal.com/55324.html

The guy writing that post knows how to use SELinux and was on a recent version of Fedora, yet he still spent several minutes getting a boarding pass to print. He fixed his problem by deciphering a log message that only someone that already understood SELinux could get anything from, using some SELinux-specific tools, and then writing SELinux policy configuration. An average sysadmin would have probably been screwed.

From 82.197.216.4 at 2013-06-17 06:34:21:

The issue is that a company should invest in security and in education of its personnel.

The problem is that most companies aren't known for being ethical.

So just give your boss what he pays for, or if you cannot do that in good conscience find a boss with sense of responsibility/ethics.

But yes, i still think ethics is the main issue here.

If you want to use SELinux in a meaningful way, then yes, you need to know a bit about the technical aspect of SELinux, but not only that, you will also need to know a bit about computer security in general. How much you need to know depends on your role.

A sysadm only needs to know some operational basics. A secadmin needs to know how to implement the security policy, and security officer needs to be able to design a security policy.

But sure, since many companies are greedy and unethical, they will probably expect an underpaid sysadm to be both secadm as well as security officer, and that is obviously unrealistic, but that is not SELinuxs' fault. Blame your employer.

Written on 04 June 2013.
« Security is not the most important thing to most people
The case against blog sidebars »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jun 4 01:58:50 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.