Systemd (v237) can do quite odd things with
/etc/fstab bind mounts
Suppose, not entirely hypothetically, that you have an Ubuntu 18.04
system (which has systemd v237) and you want to add a bind mount for a ZFS filesystem; for example, you want to bind
/cs/mail on to
/var/mail. You're managing ZFS filesystems
themselves through ZFS's usual boot time mechanisms
/etc/fstab, but putting the bind mount itself in
/etc/fstab is the simplest and shortest way. So you put the
/cs/mail /var/mail none bind
If you boot the system and do a '
df /var/mail', everything will
look good. But this is probably an illusion that will be exposed
if you do '
grep /var/mail /proc/mounts', because you most likely
really have two bind mounts, one on top of the other:
/dev/sda1 /var/mail ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 tank/cs/mail /var/mail zfs rw,nosuid,nodev,noatime,xattr,noacl 0 0
The ultimate cause of this is the same as it was in 2014, when I wrote
an entry on bind mounts with systemd and non-
fstab filesystems. Since there is no
/etc/fstab line or
/cs/mail, systemd doesn't know that it's a separate
filesystem until ZFS's commands run and the filesystem magically
appears. So apparently systemd does the bind mount twice, once when just
the root filesystem is mounted (which looks like everything needed) and
then a second time when a
/cs/mail filesystem appears and it knows
more. If another program (such as your IMAP server) looks at
before the second mount appears, it will see (and possibly take a
reference to) the bad, empty version on the root filesystem.
The automatically created systemd mount unit for
include a '
/cs/mail', but this doesn't mean that systemd will require a
/cs/mail mount. When systemd starts, as far as it knows
only requires the root filesystem.
Suppose that you then decide to get clever and use brute force. If
systemd is bind mounting the root filesystem's empty
directory, let's give it a mount source that doesn't exist on the
root filesystem by changing
fstab to be:
/cs/mail/mail /var/mail none bind
Your ZFS filesystem remains
/cs/mail; you will just put everything
in a directory in it,
/cs/mail/mail. So you make this change,
reboot your system, and now you find that
/cs/mail is not mounted
as a ZFS filesystem, and you still have a bind mount from the root
filesystem. As far as I can tell, systemd v237 will automatically
mkdir the source of a bind mount if it doesn't exist. So systemd
/cs/mail/mail on the root filesystem and bind mounted it to
/var/mail, and then ZFS found that its
/cs/mail mount point wasn't
empty and refused to mount the ZFS filesystem. My clever idea made me
worse off than before.
(This behavior doesn't seem to be documented in the current systemd.mountd(5) manual page, but systemd v237 does date from 2018. On the other hand, it's not documented in the v237 version of the manual page either.)
These days, systemd allows you to express some but not all systemd
unit conditions in
/etc/fstab (see [systemd.mountd(5)]]). However,
I still think that a real
.mount unit file is your most reliable
bet. This also lets you use '
to check for the presence of some known file or subdirectory in
your original filesystem in a way that hopefully won't cause systemd
to create it on the root filesystem if it's missing.
(In a modern systemd you can also use
but that's not in v237.)
PS: I haven't tested this on anything more recent than the Ubuntu 18.04 systemd, since Ubuntu 18.04 is what we're going to be using for the particular server where this is an issue (because it's what all our other ZFS fileservers use). That is also part of why this is not currently any sort of bug report to anyone. For that matter, I don't know if this is a bug, other than perhaps a documentation bug.