An odd error I encountered with ZFS snapshots on Ubuntu 18.04
Recently I was looking at something in a snapshot on one of our Linux ZFS fileservers and encountered a very odd error message:
# cd /h/400/.zfs/snapshot # ls [...] # cd somename/fred -bash: cd: somename/fred: Object is remote # cd somename -bash: cd: somename: No such file or directory
Examination showed that I had misspelled the name of the snapshot I
was trying to enter, but this 'Object is remote' error message was
one I didn't think I've ever seen before. The specific errno is
EREMOTE, which errno(3) says is
what the error text here says. On x86 Linux, this is errno 66:
$ errno -s remote EREMOTE 66 Object is remote EREMCHG 78 Remote address changed EREMOTEIO 121 Remote I/O error
You won't see this error message from a more recent Linux kernel. These days you'll get the usual 'no such file or directory' error (at least with more recent versions of OpenZFS; I haven't tried it with older ones).
The cause of this peculiar error is a specific behavior in older Linux kernels. Internally, the .zfs/snapshot directory is an automounter directory (with each snapshot as a new sub-filesystem). If you try to access a snapshot that doesn't exist, ZFS returns a 'EISDIR' error to the Linux kernel's VFS, and then the VFS decides, well, let me quote a comment from old kernel source:
The filesystem is allowed to return -EISDIR here to indicate it doesn't want to automount. For instance, autofs would do this so that its userspace daemon can mount on this dentry.
However, we can only permit this if it's a terminal point in the path being looked up; if it wasn't then the remainder of the path is inaccessible and we should say so.
The 'saying so' the kernel does here is returning an
error up to user level. The kernel only does this if you're trying
to access something under the bad snapshot name (ie, if the snapshot
name isn't the terminal point in the path); if you're just trying
to access the snapshot itself, the VFS doesn't transmogrify it this
You won't get this error message today because the code involved was removed from the Linux kernel in version 5.6, specifically in this commit by Al Viro. If you look at the diffs, you can see the entire comment I quoted above and the relevant code below it get removed from fs/namei.c. Since Ubuntu 20.04's default kernel is based on 5.4, you can probably see it there, but not on Ubuntu 22.04.
(How I found this commit is that I have a copy of the Ubuntu 18.04
kernel tree, so I searched the fs/ area for EREMOTE. Once I'd found
the relevant bit in fs/namei.c, I went to my copy of the main Linux
kernel tree and did '
git log -G EREMOTE fs/namei.c', after scanning
the git-log manual page because
I had a vague memory that there was a way to search for strings in