2008-03-22
Why NFS writes to ZFS are sometimes (or often) slow
It's a relatively well known issue that writing lots of small files over NFS to a ZFS filesystem is slow, but I was surprised to discover that it had a significant slowdown even when doing large bulk streaming writes to single files. Discovering this got me curious enough to dig into things.
Like most recent filesystems, ZFS is a journaled, using what the ZFS people call the ZIL (ZFS Intent Log). Also like other journaled filesystems, ZFS has the fsync problem. So where do the syncs come from?
The first version of NFS required all writes to be synchronous, with the
server not allowed to reply to them until the data was on disk, which
was soon widely acknowledged as a terrible idea for performance. NFS
v3 fixed this by allowing asynchronous writes and introducing a new
operation, COMMIT, to force the server to flush some of your async
writes to disk. If the server can't do this, for example because it has
rebooted and lost some of your async writes, it will tell you and it's
your obligation to resend the writes.
NFS v3 COMMITs are a form of fsync()s, and so they force ZFS to flush
the ZIL, with the resulting performance hit. One of the times that NFS
v3 clients send a COMMIT is when you close() a file, which is why
writing lots of small files is slow on ZFS; there's an expensive sync
after every file.
What is going on with large files is the corollary of async writes and
COMMIT: if you have not COMMITed a range of writes, the server is
free to lose them. Which means that you must be able to resend those
writes, and thus have to keep the data sitting around in your writeback
cache until you get a positive reply to your COMMIT. Thus, every so
often the client has to send a COMMIT to the NFS server so that it can
free up some of its writeback cache.
(Indeed, this is what I see when looking at NFS server stats; there are several hundred COMMITs over the course of writing a 10 GB file.)
All of this says nothing about whether the NFS write slowdown actually matters to you; that's something that depends on your usage patterns and what sort of performance you need. The performance I've measured in our test environment, while not stellar, is probably good enough for us.