What I want to know about kernel security updates
January 23, 2013
This is kind of a rant. The issue is on my mind because we spent a chunk of this evening applying kernel updates to our Ubuntu machines and rebooting them, something that we feel forced to do once every few months or so. One of the reasons that we don't do this more often, such as every time when Ubuntu releases a kernel update, is that kernel updates are among the most disruptive updates that there are; in order to make them take effect you must reboot the machine, which is completely disruptive to anyone using the machine (especially if they're logged in to it).
But another reason we don't apply Ubuntu kernel updates all that often is that Ubuntu's kernel updates are terrible at giving us useful information about how severe the issues are and how urgent doing an update is. Except in terribly obvious extreme situations (eg 'locally exploitable bug, gives root, an exploit is public') we wind up faced with a flurry of issues of extremely uncertain but generally low seeming impact. Unsurprisingly we wind up defaulting to not doing major disruptions on a regular basis, then periodically we decided that we should get up to date just in case.
While Ubuntu has its specific failings here, this is not just an Ubuntu problem. I think every Linux distribution I've seen a kernel security update from has failed to include the information we'd need to make meaningful decisions. All of them irritate me.
As a sysadmin, here is what I want to know about every issue fixed in a kernel security update:
I suspect that most distributions would want to put together their own information page in some standardized format. This is fine, just as long as they put a link to their own info page in the announcement and their info page links to the primary source (and the CVE information and so on). This would also be a good place to put extended discussions of things like how to tell if your particular system is potentially vulnerable to the issue.
My excessively cynical side suspects that distribution security teams leave out a lot of this information in order to push people towards applying every kernel update as soon as possible. If so, I have news for those security teams: they have it exactly backwards. There are powerful forces pushing us (and anyone) against applying updates, especially disruptive updates like new kernels. Every doubt and quibble and uncertainty in a kernel update message feeds those forces and makes it less likely that the update will be applied. In order to get us to apply an important update on an urgent basis, it must be clear that it is urgent. If it is not clear, everyone loses.
Everything works much better when the security team is honest and clear about kernel updates. We'll still sit on all of the updates that are just yet more ways for local users to lock the machine up, but that's no different than what we're already doing. But when you release something that's genuinely dangerous we'll be much more likely to notice, understand, and update much earlier than we would otherwise.
(By the way, for everyone who is about to advise us that we should have dynamic load balancers and pools of machines where we can take some out of service on a rolling basis for kernel upgrades and so on: there is no such thing as a general dynamic load balancer for user login sessions, established sessions in general, or actual running user processes. Thanks.)
Written on 23 January 2013.
* * *