Wandering Thoughts archives

Some bits on Linux NFS(v3) server filesystem IDs (and on filehandles)

NFS(v3) filehandles are how NFS clients tell the NFS server what they're operating on, and one part of the filehandle is a 'fsid', a filesystem ID (or UUID). The Linux kernel NFS server normally automatically determines the fsid to use itself, but you can explicitly set it in your NFS exports, per exports(5), and doing so may let you move filesystems from one actual server to another without clients having to unmount and remount the filesystem (as I recently discovered we needed to do in one ZFS fileserver upgrade situation). This makes the question of what the fsid of your NFS exported filesystems a matter of some potential interest, and also where it comes from.

Based on a comment from Arnaud Gomes on yesterday's entry, the answer to where you can observe the fsid of existing NFS exports turns out to be /proc/fs/nfsd/exports. Filesystems only show up here once someone actually mounts them, but when they do you'll get lines like this (on Ubuntu 22.04):

# Version 1.1
# Path Client(Flags) # IPs
/w/435  @nfs_ssh(rw, root_squash, sync, wdelay, no_subtree_check, uuid=7341d08c:00034ae1:00000000:00000000, sec=1)

(Except with no spaces after the commas, I put them in to allow your browser to wrap the line.)

Similar contents can be found in /proc/net/rpc/nfsd.export/content, if you prefer looking there instead. In this output, I believe the 'uuid' field is the fsid, although I don't know if exportfs will accept this name for it or if you have to write it as 'fsid=...' in your exports.

(I believe that a plain numeric 'fsid' is quite different from a UUID here; in fact, I believe they'll generate completely different NFS filehandles. You really need to specify the exact UUID and have the kernel accept it as a UUID in order to get the same NFS filehandle.)

Based on a quick look at the kernel source, it appears that there are a number of different types of fsids. To find out what type your particular filesystem and mount point is using, you can look in /proc/net/rpc/nfsd.fh/content:

#domain fsidtype fsid [path]
@nfs_ssh 6 0x8cd04173e14a03000000000000000000 /w/435

The definitions of the fsidtype numbers are in fs/nfsd/nfsfh.h, and here '6' is 'FSID_UUID16', a 16-byte UUID. As we see here, this UUID appears to be flipped around from the version listed in exports in a somewhat complex way. In full filehandles, there are also different types of 'fileid', which are covered in include/linux/exportfs.h. At the moment, ZFS on Linux appears to use only 'FILEID_INO32_GEN' ('1'), which has a 32-bit inode number and a 32-bit generation number.

(If you simply specify a 'fsid=' plain number in your exports, I suspect you get a fsidtype of 'FSID_NUM', aka 1.)

With ZFS (and probably other filesystems), if you export a subdirectory of a filesystem instead of the root of the filesystem, what you get is a fsidtype of 7, 'FSID_UUID16_INUM', which is the 16-byte UUID plus an 8-byte inode number. The 8-byte inode number appears on the front of the UUID in /proc/net/rpc/nfsd.fh/content and appears to be the visible inode number that 'ls -ldi' will tell you.

As documented in nfsd(7), it's possible to use a special interface in /proc/fs/nfsd to get the full filehandle for a given file in an exported filesystem, using the 'filehandle' file. I'll show an example in interactive Python:

>>> f = open("filehandle", mode="r+")
>>> f.write("@nfs_ssh /w/435/cks 128\n")
>>> r = f.read()
>>> print(r)
\x01 00 06 01 7341d08c 00034ae1 00000000 00000000 0a000200000000001d000000

The actual output has no spaces, but I've broken it up to show some of the observable structure, which comes from fs/nfsd/nfsfh.h. This is version 1, an auth type that is ignored and may always be 0, a type 6 fsid, a type 1 fileid, the 16-byte filesystem UUID as shown in the same format as in the exports, and then a blob that I'm not going to try to decode into its component parts because that would require too much digging in the ZFS code.

(Interested parties can start with the fact that the observable inode number for this directory is '2'.)

You get an interestingly different filehandle for the root of the exported filesystem. I'll show it decoded again:

\x 01 00 06 00 7341d08c 00034ae1 00000000 00000000

This is still a type 6 fsid, but now the fileid is 'FILEID_ROOT' (0), the root of the exported filesystem. Since the root is unique, we only have the filesystem UUID; there's no extra information. Well, more exactly we probably have the filesystem 'fsid' in the sense of /proc/net/rpc/nfsd.fh/content, which here is the filesystem UUID.