Today I used bind mounts in an odd way:

Today's crazy Linux bind mount usage:
# cp [-a] /a/nfs/mount/special-file /root/special-file
# mount --bind /root/special-file /a/nfs/mount/special-file

This was the easiest way to make a systemd service with DynamicUser=yes and a supplementary group get access to special-file, which is only accessible by said group. (The normal version of the service runs with the file not on NFS.)

I assume something about filesystem visibility for systemd dynamic users but meh, life is short and I have a hammer.

(Oops, I see I left out a critical '-a' cp argument in my initial Fediverse post, cf.)

Linux's bind mounts are normally used with directories, but it's equally valid to bind mount a file, as I'm doing here. By bind-mounting the special file the service needs to access to a local file, I'm taking NFS out of the picture. This turned out to be the right answer (and in fact the only good one), but not for the reasons that I thought.

This particular service uses DynamicUser=yes because it's a combination of more convenient and more secure. Because things run by the service need to read a private file, the service also has a supplementary group; the private file is in the group, and the service has access to the group. In the production deployment, this file lives on a local filesystem; here, I was running a test setup, where having it on NFS is more convenient. At first, I assumed that DynamicUser=yes was manipulating NFS mount related things so that the supplementary group was ignored (it wasn't completely blocking NFS mount access, because other things the service was using came from the same NFS mount), but this isn't the problem. Instead, the problem is on our Linux NFS servers.

Like many other people, our Linux NFS servers are configured to allow people to (meaningfully) be in more than 16 groups, which is the NFS v3 protocol limit. On Linux NFSv3 servers, how this works is that the NFS server throws away the group list from the NFS client and does its own local lookup. We have a synchronized password file, so for regular logins and groups the NFS servers have the same UIDs and GIDs as the NFS clients (including for the supplemental group used here) and this all works out. However, when you set DynamicUser=yes, systemd makes up a new UID (and GID) that doesn't exist in your local /etc/passwd and so won't exist in the NFS server's /etc/passwd either. When a process in the service makes NFS requests, the NFS server takes the carefully curated list of supplemental groups you set up in systemd, throws them away, looks up the UID in its own /etc/passwd and /etc/group, finds nothing, and concludes that this request has no group permissions at all.

(Indeed, now that I look I can see the telltale '<uid> 0:' line in the NFS server's /proc/net/rpc/auth.unix.gid/content, cf. Along with a few other unknown UIDs that we're seeing from somewhere.)

When I used a bind mount to make the special file a local file, not a NFS file, I bypassed the NFS server and with that, the NFS server ignoring the local supplemental group. Once all of the access control for the file was being done locally, by the client's kernel, the supplemental group worked to allow access. I believe this was the only way to solve the problem without changing the service unit.

So the end moral is supplemental groups don't work over NFSv3 with systemd dynamic users. More generally, supplemental groups with anonymous UIDs don't work over NFS; systemd dynamic users are merely one way to get anonymous UIDs. For our uses this isn't a fatal problem, but I'll want to remember it for the future.

(The workaround would be to allocate an actual UID for this purpose, set it in the systemd unit file, and then possibly duplicate all of the additional things that DynamicUser=yes normally does that increase security and isolation.)

(I realized what the answer was due to a suggestion from Steven Reid.)