2016-05-23
How fast fileserver failover could matter to us
Our current generation fileservers don't have any kind of failover system, just like our original generation. A few years ago I wrote that we don't really miss failover, although I allowed that I might be overlooking situations where we'd have used failover if we had it. So, yeah, about that: on reflection, I think there is a relatively important situation where we could really use fast, reliable cooperative failover (when both the old and new hosts of a virtual fileserver are working properly).
Put simply, the advantage of fast cooperative failover is that it makes a number of things a lot less scary, because you can effectively experiment (assuming that the failover is basically user transparent). For instance, trying a new version of OmniOS in production, where it's very unlikely that it will crash outright but possible that we'll experience performance problems or other anomalies. With fast failover, we could roll a virtual fileserver on to a server running the new OmniOS, watch it, and have an immediate and low impact way out if something comes up.
(At one point this would have made our backups explode because the backups were tied to the real hosts involved. However we've changed that these days and backups are relatively easy to shift around.)
It's possible that we should take another look at failover in our current environment, since a lot of water has gone under the bridge since we last gave up on it. This also sparks a more radical thought; if we're going to use failover mostly as a way to do experiments, perhaps we should reorganize things so that some of our virtual fileservers are smaller than they are now so moving one over affects fewer people. Or at least we could have a 'sysadmin virtual fileserver' so we can test it with ourselves only at first.
(Our current overall architecture is sort of designed with the idea that a host has only one virtual fileserver and virtual fileservers don't really share disks with other ones, but we might be able to do some tweaks.)
All of this is a bit blue sky, but at the very least we should do a bit of testing to see how much time a cooperative fileserver failover might take in our current environment. I should also keep an eye out for future OmniOS changes that might improve it.
(As usual, re-checking one's core assumptions periodically is probably a good idea. Ideally we would have done some checking of this when we were initially testing each OmniOS version, but well. Hopefully next time, if there is one.)
2016-05-16
Discovering my personal limit on how much I care about security
Sometimes finding out about hardware and software cause you to find out things about yourself too. In my case, discovering and reading up on two-factor authentication tokens that you can use to hold your SSH keys (as I more or less want) led me to discover the limits of how much I care about my personal security.
To be specific, it turns out that I don't care enough to pay for a 2FA token for personal use (cf). It's not that they're particularly expensive (although you may not want a current generation Yubikey), but even that modest expense is enough that my inaction shows that I've passively decided that I don't care that much.
Part of this is certainly that I don't think it would get me that much more security or convenience. In security, I already hold my personal SSH keys encrypted and load them into ssh-agent instead of using them directly in ssh; in convenience, well, I make them accessible in ssh-agent, so as long as I'm actively using one of my primary machines I can ssh to other machines without any challenges being involved. Putting SSH keypairs in a physical token of some sort would make them less accessible to an attacker who could compromise my workstation, but in practice if an attacker compromises my workstation I've already lost.
But that's only part of it, because if I was given a 2FA device I'd certainly go to the effort of setting it up and using it (and I'd be happy if work bought one or more to experiment with). I just don't care enough to spend any of my own money on improving my security. So that's the limit of how much I care about my security right now; free is fine, money is out.
(I'm aware that 2FA tokens can increasingly be used for things like Github and other websites in addition to just SSH logins, but I don't care enough about them either. If anything I care less about Github than I do about SSH logins, because of what I consider my Github account to be about (which is another entry entirely). Possibly I should care more and maybe someday I will, but not at the moment. I'd probably have to write something that people came to value and depend on before that happened.)
Sidebar: My view on 2FA token usage on untrusted machines
One nominal usage case for 2FA tokens is when you want to log in to your account from a machine that you don't trust enough to use a regular password from. My view there is that I don't trust other machines enough to expose a login session to them, because basically anything that I do or see in one is potentially sensitive. If I don't trust your machine not to be keylogging my password, I don't trust it to not be logging my entire SSH session.
(And ditto for things like IMAP access or using a sensitive website, although we have a few websites that I might trust a machine enough to do one time operations on. Fortunately this doesn't come up in the work that I do in specific, because I don't really have to interact with other people's machines to do things like get them on the network.)
2016-05-11
We're never going to be able to have everyone use two factor authentication
Every so often I think about two-factor authentication here or talk to people about it. In the process of doing this, I've come around to an important realization:
We're never going to be able to have all our users use two factor authentication.
The largest fundamental issue is cost. If we require universal two factor authentication, the department needs to provide 2FA tokens to everyone (and then manage them, of course). Current typical prices are $50 US one time costs plus we pay for all management, or $33/year (and someone else worries about management). At that cost level, we're looking at tens of thousands of dollars of cost. The budget for that is simply not there. Even with probably unpopular moves like charging graduate students a '2FA token deposit' or the like, we still have a not insignificant population that the department would have to cover the costs for directly (either directly out of base budget or by forcing professors to pay for postdocs, visitors, etc who are associated with them).
(We cannot assume that all people have smartphones, either, and delegate our 2FA authentication to smartphone apps.)
I presented this as a pure cost issue, but of course it's more than that; it's a cost versus benefits tradeoff here. If we were guarding extremely important data or systems it might be a cost that the department either was willing to incur or had no choice about, but the blunt reality is that we're not. And in practice moving everyone to 2FA would provide only a modest increase in security over what we currently have, since (as far as we know) account credentials are not getting compromised left and right. With high expenses and only a modest security increase, the tradeoff is not in favour of universal 2FA.
This doesn't mean that we'll never have any 2FA. Instead, what it means is that any two factor authentication deployment is going to be a selective one, where some accounts will be protected by it but many others won't be. A selective deployment puts various constraints on what software we can use and how we can deploy things. It also suggests that we may want to be able to use more than one 2FA system. Some people are likely to already have central university-issued 2FA tokens, some people will have smartphones that they can use for 2FA, and some people may have locally purchased 2FA tokens like Yubikeys. It would be good if we could accommodate them all, although this may not be realistic for various reasons.