iSCSI versus NFS

July 25, 2010

These days, an increasing number of storage appliances can do both NFS fileservice and iSCSI (generally using the same underlying pool of disk space), which has resulted in me seeing an increasing number of people who are wondering which one they should use.

The summary of my answer is that iSCSI is a SAN technology and NFS is a fileservice technology. If you want to add storage to a single machine, iSCSI will work acceptably well; if you want to share files among a bunch of machines, you want NFS. If you just want a single machine to have access to a filesystem to store files, I still think that NFS is better.

(One wild card in this is your storage appliance's management features, like snapshots, quotas, and so on, which may well differ significantly between iSCSI and NFS.)

Like all SAN technologies, iSCSI itself won't let you share a disk between multiple client systems; if you need that, you'll need to layer some sort of cluster filesystem on top. At that point you're almost certainly better off just using NFS unless you have some compelling reason otherwise. Hence NFS is the right answer for sharing files between multiple client machines.

(I find it hard to believe that iSCSI from a storage appliance plus a cluster filesystem running on the clients will have any sort of performance or management advantage over NFS from the storage appliance, but I've been surprised before. If the storage appliance's NFS server is terrible but its iSCSI target is good, the simpler solution is to have a single client machine be an NFS server for the storage.)

If all you want is ordinary file service for a single machine, I think that NFS is a better answer because it is generally going to be simpler and more portable. With NFS you can expand to giving multiple machines access to the files (even read-only access) and any machine that can speak NFS can get at the files. With iSCSI, you are pretty much locked to a single machine at a time, and you need a machine that both talks iSCSI and understands the filesystem and disk partitioning being used on the iSCSI disk; in many cases this will restrict you to a single operating system.

(There are cases, such as virtualization hosts, where your client machines are going to be doing exclusive access and really want to be dealing with 'real' devices, and having them use NFS would just result in them faking it anyways by, eg, making a single big file on the NFS filesystem. In that sort of situation I think it makes sense to use iSCSI instead of NFS for the single machine access case.)

It is tempting to say that iSCSI is better because it lets the client treat the storage like any other physical device, without having to worry about all of the networking issues that come up with NFS. This is a mistake; your iSCSI disks are running over a network and thus all of the networking issues are still there, they have just been swept under the rug and made more or less inaccessible. Pretending that they are not there does not make them go away, and in fact the history of networking protocols has shown over and over again that pretending the network isn't there doesn't work in the long run.

(Consider the history of RPC protocols that attempt to pretend that you're just making a local function call. Generally this doesn't go well, especially once latency and network glitches and so on come up. Things happening over a network have failure modes that rarely or never come up for purely local actions.)


Comments on this page:

From 125.255.117.141 at 2010-07-25 02:16:27:

One advantage of iSCSI if you decide to use it as a share for multiple machines: whatever file sharing system you layer over the top probably has saner file locking semantics than NFS.

By cks at 2010-07-25 23:12:35:

I honestly don't think that NFS's file locking semantics are particularly bad, assuming that you are willing to rely on actual (Unix) locks. Most of the problems tend to be issues of implementation quality, and a cluster filesystem can fumble that one too.

Written on 25 July 2010.
« Thinking about the implications of your program being successful
Why sysadmins almost never replace distribution packages »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jul 25 00:40:55 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.