Some bits on Linux NFS(v3) server filesystem IDs (and on filehandles)

March 10, 2023

NFS(v3) filehandles are how NFS clients tell the NFS server what they're operating on, and one part of the filehandle is a 'fsid', a filesystem ID (or UUID). The Linux kernel NFS server normally automatically determines the fsid to use itself, but you can explicitly set it in your NFS exports, per exports(5), and doing so may let you move filesystems from one actual server to another without clients having to unmount and remount the filesystem (as I recently discovered we needed to do in one ZFS fileserver upgrade situation). This makes the question of what the fsid of your NFS exported filesystems a matter of some potential interest, and also where it comes from.

Based on a comment from Arnaud Gomes on yesterday's entry, the answer to where you can observe the fsid of existing NFS exports turns out to be /proc/fs/nfsd/exports. Filesystems only show up here once someone actually mounts them, but when they do you'll get lines like this (on Ubuntu 22.04):

# Version 1.1
# Path Client(Flags) # IPs
/w/435  @nfs_ssh(rw, root_squash, sync, wdelay, no_subtree_check, uuid=7341d08c:00034ae1:00000000:00000000, sec=1)

(Except with no spaces after the commas, I put them in to allow your browser to wrap the line.)

Similar contents can be found in /proc/net/rpc/nfsd.export/content, if you prefer looking there instead. In this output, I believe the 'uuid' field is the fsid, although I don't know if exportfs will accept this name for it or if you have to write it as 'fsid=...' in your exports.

(I believe that a plain numeric 'fsid' is quite different from a UUID here; in fact, I believe they'll generate completely different NFS filehandles. You really need to specify the exact UUID and have the kernel accept it as a UUID in order to get the same NFS filehandle.)

Based on a quick look at the kernel source, it appears that there are a number of different types of fsids. To find out what type your particular filesystem and mount point is using, you can look in /proc/net/rpc/nfsd.fh/content:

#domain fsidtype fsid [path]
@nfs_ssh 6 0x8cd04173e14a03000000000000000000 /w/435

The definitions of the fsidtype numbers are in fs/nfsd/nfsfh.h, and here '6' is 'FSID_UUID16', a 16-byte UUID. As we see here, this UUID appears to be flipped around from the version listed in exports in a somewhat complex way. In full filehandles, there are also different types of 'fileid', which are covered in include/linux/exportfs.h. At the moment, ZFS on Linux appears to use only 'FILEID_INO32_GEN' ('1'), which has a 32-bit inode number and a 32-bit generation number.

(If you simply specify a 'fsid=' plain number in your exports, I suspect you get a fsidtype of 'FSID_NUM', aka 1.)

With ZFS (and probably other filesystems), if you export a subdirectory of a filesystem instead of the root of the filesystem, what you get is a fsidtype of 7, 'FSID_UUID16_INUM', which is the 16-byte UUID plus an 8-byte inode number. The 8-byte inode number appears on the front of the UUID in /proc/net/rpc/nfsd.fh/content and appears to be the visible inode number that 'ls -ldi' will tell you.

As documented in nfsd(7), it's possible to use a special interface in /proc/fs/nfsd to get the full filehandle for a given file in an exported filesystem, using the 'filehandle' file. I'll show an example in interactive Python:

>>> f = open("filehandle", mode="r+")
>>> f.write("@nfs_ssh /w/435/cks 128\n")
>>> r = f.read()
>>> print(r)
\x01 00 06 01 7341d08c 00034ae1 00000000 00000000 0a000200000000001d000000

The actual output has no spaces, but I've broken it up to show some of the observable structure, which comes from fs/nfsd/nfsfh.h. This is version 1, an auth type that is ignored and may always be 0, a type 6 fsid, a type 1 fileid, the 16-byte filesystem UUID as shown in the same format as in the exports, and then a blob that I'm not going to try to decode into its component parts because that would require too much digging in the ZFS code.

(Interested parties can start with the fact that the observable inode number for this directory is '2'.)

You get an interestingly different filehandle for the root of the exported filesystem. I'll show it decoded again:

\x 01 00 06 00 7341d08c 00034ae1 00000000 00000000

This is still a type 6 fsid, but now the fileid is 'FILEID_ROOT' (0), the root of the exported filesystem. Since the root is unique, we only have the filesystem UUID; there's no extra information. Well, more exactly we probably have the filesystem 'fsid' in the sense of /proc/net/rpc/nfsd.fh/content, which here is the filesystem UUID.


Comments on this page:

By cks at 2023-03-15 12:23:52:

I don't think these system calls generate a NFS filehandle, just an internal kernel handle for the file that can be used to refer to it later. Why I believe this is that a NFS filehandle must be generated in the context of a specific 'client' (ie, a set of export permissions), because the NFS fsid or uuid is specified in them, and these functions don't take a NFS client identifier.

As we see here, this UUID appears to be flipped around from the version listed in exports in a somewhat complex way.

Middle-endian in the wild! Not bad, not bad.

name_to_handle_at(open("/path_to_NFS_file"), "", ..., AT_EMPTY_PATH) returns an honest-to-God NFS file handle for the file. It was a very long time ago when I implemented NFS support for a Linux file system (reiserfs), but file handles are not client-dependent, you can in fact copy them between clients.

And lo and behold, it is still here and still breaks permission checks (linux/fs/nfs/export.c):

/*

* Let's break subtree checking for now... otherwise we'll have to embed parent fh
* but there might not be enough space.
*/

static int nfs_encode_fh(struct inode *inode, __u32 *p, int *max_len, struct inode *parent)

By cks at 2023-03-17 22:21:15:

I've checked, and indeed the NFS server code can give different filehandles for different clients, as far as I can see. I don't know if the NFS server verifies that filehandles from the client are right for the client or just valid in general, though.

(And sane configurations won't have different filehandles for different clients; you only get it under situations that are probably a mistake.)

Written on 10 March 2023.
« ZFS on Linux and when you get stale NFSv3 mounts
As a system administrator, I work in many different environments »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Mar 10 23:03:49 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.