A gotcha with Systemd's
DynamicUser, supplementary groups, and NFS (v3)
Today's crazy Linux bind mount usage:
# cp [-a] /a/nfs/mount/special-file /root/special-file
# mount --bind /root/special-file /a/nfs/mount/special-file
This was the easiest way to make a systemd service with DynamicUser=yes and a supplementary group get access to special-file, which is only accessible by said group. (The normal version of the service runs with the file not on NFS.)
I assume something about filesystem visibility for systemd dynamic users but meh, life is short and I have a hammer.
(Oops, I see I left out a critical '-a' cp argument in my initial Fediverse post, cf.)
Linux's bind mounts are normally used with directories, but it's equally valid to bind mount a file, as I'm doing here. By bind-mounting the special file the service needs to access to a local file, I'm taking NFS out of the picture. This turned out to be the right answer (and in fact the only good one), but not for the reasons that I thought.
This particular service uses
because it's a combination of more convenient and more secure. Because things run by the service need
to read a private file, the service also has a supplementary group;
the private file is in the group, and the service has access to the
group. In the production deployment, this file lives on a local
filesystem; here, I was running a test setup, where having it on
NFS is more convenient. At first, I assumed that
was manipulating NFS mount related things so that the supplementary
group was ignored (it wasn't completely blocking NFS mount access,
because other things the service was using came from the same NFS
mount), but this isn't the problem. Instead, the problem is on our
Linux NFS servers.
Like many other people, our Linux NFS servers are configured to
allow people to (meaningfully) be in more than 16 groups, which is
the NFS v3 protocol limit. On Linux
NFSv3 servers, how this works is that the NFS server throws away
the group list from the NFS client and does its own local lookup. We have a synchronized password file,
so for regular logins and groups the NFS servers have the same UIDs
and GIDs as the NFS clients (including for the supplemental group
used here) and this all works out. However, when you set
DynamicUser=yes, systemd makes up a new UID (and GID) that
doesn't exist in your local /etc/passwd and so won't exist in the
NFS server's /etc/passwd either. When a process in the service makes
NFS requests, the NFS server takes the carefully curated list of
supplemental groups you set up in systemd, throws them away, looks
up the UID in its own /etc/passwd and /etc/group, finds nothing,
and concludes that this request has no group permissions at all.
(Indeed, now that I look I can see the telltale '<uid> 0:' line in the NFS server's /proc/net/rpc/auth.unix.gid/content, cf. Along with a few other unknown UIDs that we're seeing from somewhere.)
When I used a bind mount to make the special file a local file, not a NFS file, I bypassed the NFS server and with that, the NFS server ignoring the local supplemental group. Once all of the access control for the file was being done locally, by the client's kernel, the supplemental group worked to allow access. I believe this was the only way to solve the problem without changing the service unit.
So the end moral is supplemental groups don't work over NFSv3 with systemd dynamic users. More generally, supplemental groups with anonymous UIDs don't work over NFS; systemd dynamic users are merely one way to get anonymous UIDs. For our uses this isn't a fatal problem, but I'll want to remember it for the future.
(The workaround would be to allocate an actual UID for this purpose,
set it in the systemd unit file, and then possibly duplicate all of
the additional things that
DynamicUser=yes normally does that
increase security and isolation.)