Wandering Thoughts archives

I think I like systemd's DynamicUser feature (under the right circumstances)

Our Prometheus metrics system involves a lot of daemons that do things like generate metrics, both official daemons and various third party ones. Many of these daemons and the things they do are essentially stateless, because they can be and it makes them simpler. I was recently setting up such a daemon (a new one) on my office workstation, and as part of that I wanted to pick a UID for it to run as. I consider this particular daemon slightly more risky than usual, so I didn't want to go with either root (it doesn't need that much power) or the 'prometheus' user, which is not root but does wind up owning things like the Prometheus metrics data storage. At about the time I was looking through my /etc/passwd to try to find a user that I was comfortable with and that would work, a little light went on in my mind and I remembered that systemd services can use dynamic users.

Stateless daemons with no special permissions requirements are an ideal match for dynamic users, because basically I just want a generic non-root user. The user doesn't need to have any special privileges, and it doesn't need a stable UID or GID because it will never have anything on disk (or at least, not outside of brief uses of /tmp). Systemd making these UIDs and GIDs up on the fly saves me the effort of creating one or more new users, and as you can see, having to explicitly create new users is enough annoyance that I might not do it at all.

(The other advantage of dynamic users here is that if I decide to stop using a daemon, I'm not left with a stray user and group to clean up at some indefinite point in the future.)

Switching my .service file from a 'User=' line to 'DynamicUser=yes' basically just worked. After a daemon restart, the daemon was running happily under its new unique UID with everything working fine. The daemons I converted had no problems running other programs or making network connections, either.

You don't have to restrict a DynamicUser service to standard Unix 'regular UID' permissions (well, somewhat less than that, since systemd adds extra restrictions). I run Prometheus's Blackbox exporter as a non-root user but explicitly augment its capabilities with CAP_NET_RAW so that it can send and receive ICMP packets:

[Service]
[...]
CapabilityBoundingSet=CAP_NET_RAW
AmbientCapabilities=CAP_NET_RAW

This still works fine with it converted from 'User=prometheus' to 'DynamicUser=yes'.

After this positive experience, I'm probably going to start making more use of 'DynamicUser=yes'. If a stateless thing doesn't have to run as root, switching it to using a dynamic user is both pretty trivial and a bit more secure.

(Systemd theoretically supports dynamic users for services with some state, but there can be problems with that.)

ZFS pool imports happen in two stages of pool configuration processing

The mechanics of how ZFS pools are imported is one of the more obscure areas of ZFS, which is a potential problem given that things can go very wrong (often with quite unhelpful errors). One special thing about ZFS pool importing is that it effectively happens in two stages, first with user-level processing and then again in the kernel, and these two stages use two potentially different pool configurations. My primary source for this is the discussion from Illumos issue #9075:

[...] One of the first tasks during the pool load process is to parse a config provided from userland that describes what devices the pool is composed of. A vdev tree is generated from that config, and then all the vdevs are opened.

The Meta Object Set (MOS) of the pool is accessed, and several metadata objects that are necessary to load the pool are read. The exact configuration of the pool is also stored inside the MOS. Since the configuration provided from userland is external and might not accurately describe the vdev tree of the pool at the txg that is being loaded, it cannot be relied upon to safely operate the pool. For that reason, the configuration in the MOS is read early on. [...]

Here's my translation of that. In order to tell the kernel to load a pool, 'zpool import' has to come up with a vdev configuration for the pool and then provide it to the kernel. However, this is not the real pool configuration; the real pool configuration is stored in the pool itself (in regular ZFS objects that are part of the MOS), where the kernel reads it again as the kernel imports the pool.

Although not mentioned explicitly, the pool configuration that 'zpool import' comes up with and passes to the kernel is not read from the canonical pool configuration, because reading those ZFS objects from the MOS requires a relatively full implementation of ZFS, which 'zpool import' does not have (the kernel obviously does). One source of the pool configuration for 'zpool import' is the ZFS cache file, /etc/zfs/zpool.cache, which theoretically contains current pool configurations for all active pools. How 'zpool import' generates a pool configuration for exported or deleted pools is sufficiently complicated to need an entry of its own.

This two stage process means that there are at least two different things that can go wrong with a ZFS pool's configuration information. First, 'zpool import' may not be able to put together what it thinks is a valid pool configuration, in which case I believe that it doesn't even try to pass it to the kernel. Second, the kernel may dislike the configuration that it's handed for its own reasons. In older versions of ZFS (before better ZFS pool recovery landed), any mismatch between the actual pool configuration and the claimed configuration from user level was apparently fatal; now, only some problems are fatal.

As far as I know, 'zpool import' doesn't clearly distinguish between these two cases in its error messages when you're actually trying to import a pool. If you're just running it to see what pools are available, I believe that all of what 'zpool import' reports comes purely from its own limited and potentially imperfect configuration assembly, with no kernel involvement.

(When a pool is fully healthy and in good shape, the configuration that 'zpool import' puts together at the user level will completely match the real configuration in the MOS. When it's not is when you run into potential problems.)