Wandering Thoughts archives

Security is everyone's job (why Ruby is wrong about OpenSSL)

The short version of the Ruby controversy of the day is here. The really short version is that the Ruby core has bindings for OpenSSL, some versions of OpenSSL have defaults that are terrible for security, and the Ruby core developers have decided that they will not change the Ruby bindings to fix these terrible defaults. Instead they are simply washing their hands of the issue and making security the responsibility of users of the bindings (and OpenSSL).

The Ruby core developers are wrong. Security is everyone's job, all the way up and down the stack. If someone below you screws up, it is your job to fix this for your users. By rejecting this the Ruby developers have chosen mathematics instead of people, whether they understand it or not.

I don't say this as a matter of high-minded principles. I say this as a matter of pragmatic engineering because we already know what happens if we don't do this. If you leave security problems exposed by default, some number of your users will do nothing to fix them for various reasons and will themselves be insecure. Often this will be a reasonably large number of your users. If you actually care about security, this is a clearly bad thing. Conversely if you engineer things right and hide security problems by default, many of your users will use your defaults and be secure. The golden rule is that many of your users will never touch your defaults unless they clearly don't work, so they will get whatever security you left them by default.

(Nor is this a novel principle. We've experienced it over and over, across almost every conceivable environment, and not just in the context of security but the context of every single default ever. We know this.)

At this point in time, 'insecure by default' is plain and simply wrong. The inevitable and entirely predictable result of shipping an insecure by default layer is a raft of future downstream security updates as plenty of downstream developers have to individually make the fix that you could have done once, for all of them. Oh, and their software was vulnerable in the mean time.

(And if you argue that your downstream developers should all be smarter than that, as usual you are not solving the real problem. Security is people.)

Hard drives really do wear out, so you need a a hardware budget

Some people are already laughing at the title of this entry but really, hear me out. Both I individually and where I work collectively have generally had really good luck with hardware. For the most part things just haven't broken and as a result we run both servers and hard drives well beyond what people consider their normal sane lifetimes. Generally everything has kept going and going and going, which is very convenient for a place that has traditionally had essentially no hardware budget.

Our current fileservers, their iSCSI backends, and most importantly basically all of their data disks date from mid-2008 and mid-2009. These disks are consumer 7200 RPM SATA drives, although ones we bought with five year warranties. You'll notice that the mid-2008 drives are definitely out of warranty and the mid-2009 drives are approaching the edge of it. Vaguely based on general experiences we've been sort of expecting things to tick on even (well) past that five year data; sure, we ought to replace the drives on general principles but that was a vague thing.

It hasn't worked out that way. Our drives have been dropping like flies lately. A significant part of this is with the mid-2009 drives, where we had the misfortune to buy the now infamously unreliable Seagate 1TB ST31000340AS model, but even our older mid-2008 drives are starting to die at an alarming rate. And our 1TB loss rate is not really sustainable either.

(By 'dropping like flies' I mean 'we lost two drives over the past two-week Christmas break and this has stopped being at all exceptional'. We currently have 72 drives in fileserver production.)

The good and lucky news is that this summer we got a budget for replacement hardware more or less out of the blue and as a result we're well along with work to replace all of the hardware involved, most especially the disks. But that's almost pure luck because until very recently we haven't been feeling any particular sense of urgency about the replacement project. Certainly it wasn't a 'things on fire' priority project; we assumed that we had plenty of time and the whole environment was in good shape and so on.

Obviously we were wrong, very nearly disastrously so. The lesson this teaches me is that we can't get this close to the edge the next time around. We need to maintain an ongoing hardware budget (hopefully achieved) and we really do have to renew our major hardware in advance of its nominal probable end-of-life date, never mind how much luck we've had in the past with eg SATA drives in individual servers. This requires advance planning and preparation and politics and most of all it calls for making a schedule well in advance.

(It needs politics because you have to argue, in what is always a time of tight budgets, for a merely precautionary expenditure of cash. We'll be buying new hardware not because we clearly have to but because it's probably a good idea.)

So let me put it in writing: we should be turning over our fileserver disks by the end of 2018, four years after we turn them over now in 2014. That's isn't running them to the ragged edge of their warranty period (we're getting drives with five year warranties), it's 'wasting' some amount of money by replacing what will probably be perfectly operable disks before they die, but I never again want to be in a situation where I'm racing against disk failures.

(I'm currently pondering applications of this idea to my home machine, where I am partly running on a SATA drive that is over five years old. It's in a mirror pair, but still.)