2016-02-22
We've permanently disabled overlayfs on our servers
Oh look, yet another Linux kernel local exploit in the overlayfs module. Time to permanently blacklist it on all of our machines.
Today's bugs are CVE-2016-1576 and CVE-2016-1575 (via). There have been others before, and probably more that my casual Internet searches aren't turning up.
Based on my experiences so far, the two most common ingredients in exploitable kernel security issues we've been seeing Ubuntu announcements for are overlayfs and user namespaces. As far as I know, we can't do anything to turn off user namespaces without rebuilding and maintaining our own kernel packages, but overlayfs is (just) a loadable kernel module. A kernel module that we don't use.
So now we have an /etc/modprobe.d/cslab-overlayfs.conf file on
all of our servers that says:
# Permanently stop overlayfs from being loaded # because it keeps having security issues and # we don't use it. blacklist overlayfs install overlayfs /bin/false
Pretty soon this will be in our install framework, which means that future machines will probably be like this for several Ubuntu LTS versions to come. I feel some vague regret, but not very much. I'm done putting up with the whole 'surely we'll get this right someday' approach to making these subsystems not create security issues.
By the way, I don't find issues in either subsystem to be particularly surprising given what they do. User namespaces especially are a recipe for trouble in practice, because they let you create environments that break long standing Unix security assumptions. Sure, they are supposed to only do this in a way that is still secure, but in practice, no, things keep slipping through the cracks. In a sane world it would be possible to disable user namespaces at runtime on distribution kernels. Sadly we're not in that world.
The university's coordination problem
In response to my entry on how we can't use Let's Encrypt in production, Jack Dozier left a comment asking if we'd looked into InCommon's Certificate Service. InCommon is basically a consortium of US educational institutions that have gathered together to, among other things, create a flat cost CA service; apparently, for $15k US or so a year, your university can get all the certificates you want (including for affiliated organizations). This sounds great, but at least here it exposes what I'm going to call the university coordination problem.
Put simply, suppose that the university spent $15k a year to get 'all you want' certificates. More specifically, this would be the central IT services group. Now, how does the central IT group get the news out to everyone here that you can get free certificates through this program?
The University of Toronto is a big place, which means that there are a dizzying number of departments, research groups, professors, and various other people who could possibly be buying TLS certificates for something they're doing. Many of these people do not deal with IT issues like TLS certificates on an ongoing basis, so they're extremely unlikely to remember the existence of a service they might have gotten an email blast about half a year ago.
(And I guarantee that if you sent that email blast to professors, most of them deleted it unread.)
Nor is there a central place where money gets spent that you can set up as a chokepoint. I mean, yes, there is a complicated university wide purchasing department, but no one sane is going to make people get pre-approval from purchasing for, say, twenty dollar expenses. The entire university would grind to a halt if you tried that (followed immediately by a massive revolt by basically everyone). TLS certificates are well under the preapproval cost threshold, so in practice people purchase most of them through university credit cards.
In theory CAs themselves might serve as a roadblock, by requiring approval from the owner of the overall university domain. In practice I believe that many CAs will issue TLS certificates if you can simply prove ownership of the subdomain you want the certificate for. CAs have an obvious motivation to do this if they can get away with it, since it means that more people are likely to buy certificates from them.
(In general, vendors of things are highly motived to let little departments and groups buy things without the involvement of any central body, because involving central things in a big company invariably slows down and complicates the process. You really want some person in some group to just be able to put your product or service on their corporate credit card, at least initially.)
This is not an issue that's unique to TLS certificates. It's a general issue that applies to basically anything relatively inexpensive that the university might arrange some sort of a site license for. The real challenge is often not buying the site license, it's insuring that it will get widely used, and the issue there is simply getting the news out and coordinating with all of the potential users. Some products are pervasive enough or expensive enough that people will naturally ask 'do we have some sort of central licensing for this', but a lot of them are not that way. And you can be surprised about even relatively expensive products.
(For that matter, I suspect that this issue comes up for things that are expensive but uncommon. For instance, we have a site license for a relatively expensive commercial anti-spam system, but I suspect that many people running mail systems here don't know about it, even if it would be useful to them.)
PS: This problem is probably not unique to universities but is shared at least in part by any sufficiently large organization. However, I do think that universities have some features that make it worse, like less central control over money.