Wandering Thoughts archives

The basic implementation of relatively high-availability NFS

There's a lot of complicated approaches for high availability services in general. Fortunately, NFS's statelessness makes it easy to do a relatively simple, low-rent version of this.

First, make your fileserver names point to virtual IP addresses. Embody these virtual IPs on the physical machines of your choice that have some form of shared storage. To fail over a (virtual) fileserver, take down the IP address on the original machine, unshare and unmount everything, mount and share it on the new machine, and finally bring up the virtual IP address as (yet another) interface alias on the new machine.

(If the first machine crashes or otherwise goes down hard, skip the first bits. The use of various techniques to force a machine down is optional but potentially recommended.)

The one issue that we've run into with this scheme that can't be attributed to operating system bugs is that it's not clear how NFS locks get cleanly handled, especially if the original machine is hosting multiple virtual fileservers so that you can't just shoot the locking daemon and remove all of its state.

(You may also run into operating system bugs and limitations, as we have, which is why our current HA NFS server setup remains mostly theoretical.)

This is merely relatively high availability, because it may take a fair amount of time to bring the filesystems up on the new machine and export them, and then for all of the client machines to acquire the virtual fileserver's new Ethernet address (and re-establish TCP NFS connections, if you're using NFS over TCP). If you need things to go faster, you'll need to elaborate this basic outline in various ways.

(However, without a suitable cluster file system underlying your NFS fileservers I think that there's a definite maximum failover speed you'll ever achieve. If the filesystems are not live and already exported on each NFS server, you'll have to mount and export them, and that plain takes time, especially if you have a lot.)

A lament about modern NFS development

My problem with the direction that modern NFS protocol development is going is, to put it succinctly, that everyone involved seems to want to turn NFS into AFS or another real networked filesystem. Instead of just improving NFS as it stands, they seem to want to change it into something substantially different, something that would require a major restructuring of all sorts of things in order to use.

So what do I consider 'NFS as it stands'? To me, the virtues of NFS are that it is a fast, resilient system for providing networked filesystems with more or less Unix semantics between trusted hosts. While I'm biased, I happen to think that this is still an important niche; people still have lots of Unix machines within more or less the same trust domain. This isn't to say that NFS is perfect, which is part of the frustration of it all; I can think of a number of things about NFS that could stand to be improved, and most of them are untouched and going to stay that way.

(I cannot help but feel that part of the situation comes about because NFS-like filesystems are considered not modern, cool, and acceptable, so no one serious wants to be seen working on them. Comparisons to the old microkernel mania are left as an exercise to the reader.)

Sidebar: NFS improvements that I'd like to see

My off the cuff wishlist includes:

• better handling of userids, to let people relax the trust relationships a bit; this is a welcome part of NFS v4.
• automatically handle mounting and accessing sub-filesystems; I believe that this is also part of NFS v4.

• improve the host security.

• roll all of the various associated protocols and systems (mounting, lock management, etc) into the core of NFS, and make sure that they all work well together (and reliably).
• improve the recovery from server and client events. Right now the core protocol seems to recover without problems, but lock recovery can be chancy.

I don't know if NFS needs performance work to cope with future 10G networks, but I wouldn't be surprised if it did. (With decent machines and a fast enough disk system, it's already fast enough to saturate ordinary gigabit networks.)

What I want out of NFS security, at least at the moment

I know, NFS has a lot of new security features in both the more or less mythical NFS v4 and in NFS v3 with some hacks. The problem is that they all give me the wrong sort of security (as far as I can tell); like other real network filesystems, they're all focused on authenticating the users. What I want is good authentication of the hosts.

(The problem with user-based authentication is that it takes out all forms of setuid. This forces a really big, really drastic change in the fundamentals of how you structure your systems, which I am just not interested in exploring. I trust my systems, and I'd better, because they're all multi-user systems.)

The best way to authenticate hosts is for hosts to sign every NFS message, and then the other end to verify the signatures. At a minimum I'd like the host to be authenticated on each message; better would be message integrity validation as well. (Outright encryption is probably still too slow, and performance is important for me here; I can't suggest using something that cuts our NFS performance significantly.)

One can do this today with IPSec, but IPSec has two flaws for this: it's not integrated into NFS administration, and it's remarkably complex. One could probably solve both problems with a bunch of scripts, but then we'd have added another problem. And I'm don't know if current IPSec implementations can run fast enough to manage gigabit wire speeds, even just for host authentication.

(I'm also not sure if the authentication only modes are well regarded or well tested; I have the impression that most people feel that IPSec should be used with full encryption, which I believe is significantly slower than just authentication.)