An odd error I encountered with ZFS snapshots on Ubuntu 18.04

November 7, 2022

Recently I was looking at something in a snapshot on one of our Linux ZFS fileservers and encountered a very odd error message:

# cd /h/400/.zfs/snapshot
# ls
[...]
# cd somename/fred
-bash: cd: somename/fred: Object is remote
# cd somename
-bash: cd: somename: No such file or directory

Examination showed that I had misspelled the name of the snapshot I was trying to enter, but this 'Object is remote' error message was one I didn't think I've ever seen before. The specific errno is EREMOTE, which errno(3) says is what the error text here says. On x86 Linux, this is errno 66:

$ errno -s remote
EREMOTE 66 Object is remote
EREMCHG 78 Remote address changed
EREMOTEIO 121 Remote I/O error

You won't see this error message from a more recent Linux kernel. These days you'll get the usual 'no such file or directory' error (at least with more recent versions of OpenZFS; I haven't tried it with older ones).

The cause of this peculiar error is a specific behavior in older Linux kernels. Internally, the .zfs/snapshot directory is an automounter directory (with each snapshot as a new sub-filesystem). If you try to access a snapshot that doesn't exist, ZFS returns a 'EISDIR' error to the Linux kernel's VFS, and then the VFS decides, well, let me quote a comment from old kernel source:

The filesystem is allowed to return -EISDIR here to indicate it doesn't want to automount. For instance, autofs would do this so that its userspace daemon can mount on this dentry.

However, we can only permit this if it's a terminal point in the path being looked up; if it wasn't then the remainder of the path is inaccessible and we should say so.

The 'saying so' the kernel does here is returning an EREMOTE error up to user level. The kernel only does this if you're trying to access something under the bad snapshot name (ie, if the snapshot name isn't the terminal point in the path); if you're just trying to access the snapshot itself, the VFS doesn't transmogrify it this way.

You won't get this error message today because the code involved was removed from the Linux kernel in version 5.6, specifically in this commit by Al Viro. If you look at the diffs, you can see the entire comment I quoted above and the relevant code below it get removed from fs/namei.c. Since Ubuntu 20.04's default kernel is based on 5.4, you can probably see it there, but not on Ubuntu 22.04.

(How I found this commit is that I have a copy of the Ubuntu 18.04 kernel tree, so I searched the fs/ area for EREMOTE. Once I'd found the relevant bit in fs/namei.c, I went to my copy of the main Linux kernel tree and did 'git log -G EREMOTE fs/namei.c', after scanning the git-log manual page because I had a vague memory that there was a way to search for strings in the diffs.)

Written on 07 November 2022.
« Go's sync.Pool has (undocumented) 'thread' locality
Some thoughts on organizations running their own Fediverse instance »

Page tools: View Source.
Search:
Login: Password:

Last modified: Mon Nov 7 21:36:49 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.