My wish for a way to activate one systemd unit when another one started

October 16, 2021

I recently wrote (again) about bind mounts under systemd for ZFS filesystems, covering how you don't want to put them in fstab and how you might not want them to say they're required by ZFS filesystems are special here because they're not represented in /etc/fstab or in explicit .mount units; instead they unpredictably appear out of the blue when some other program is run. ZFS mounts aren't the only thing that can appear this way; our local NFS mounts are managed through a program, and for that matter removable media is not always there. These other cases can expose similar issues (especially NFS mounts). All of them could be solved through a feature that, as far as I know, systemd doesn't have, which is activating one unit when another unit asynchronously appears and becomes active.

(With such a feature, I would set bind mounts to activate when the underlying filesystem had appeared and been successfully mounted.)

Effectively what I want is something like a hypothetical 'TriggeredBy=' property. If any unit in the list appeared and became active, the unit with a TriggeredBy would automatically try to activate. Generally you'd also want an After= on the TriggeredBy units. Listed units would not have to exist at the beginning of systemd's operation, and TriggeredBy couldn't be implemented by creating /etc/systemd/system/<what>.d/ directories with drop-in files, because that doesn't work for all units (eg, the last time I looked, automatically created .mount units).

As far as I can see you can't implement this with any existing set of properties in systemd.unit(5), although maybe I'm missing something. It's possible that PartOf= and After= will do this, but if so it's not entirely clear to me and on systemd v237, this doesn't appear to work if you declare a 'PartOf=' to a ZFS filesystem's .mount unit (which will only appear to systemd when the pool is imported). Even if PartOf worked, it could be too strong for some uses (although it's probably right for a bind mount to automatically unmount if you unmount the underlying filesystem).

I understand why systemd doesn't do this. Unlike Upstart, systemd is not necessarily fundamentally built around events, so having events trigger units is not as natural a thing to it. Systemd has some limited event triggering in the form of its various activation schemes (socket, path, automounting, and timers), but all of this is specific and seems relatively hard coded. Asynchronous event handling is the domain of udev.

Comments on this page:

Would perhaps systemd.automount be the solution to this issue?

From at 2021-10-17 11:16:50:

I'm not sure I understand how this differs from WantedBy= (i.e. the .wants/ subdirectories).

As far as I've tried, it does work for nonexistent units, e.g. a nonexistent .device can trigger a .mount when it becomes existent and active (IIRC, this is how auto,nofail is implemented), and externally-done mounts do likewise pick up any foo.mount.wants/ symlinks.

Though you can't create those via systemctl add-wants, and it's possible that older systemd versions might not have allowed to systemctl enable if the WantedBy= pointed to a nonexistent unit (I haven't checked; the latest version only shows a warning), but it works fine when the symlink is done manually or through a generator.

The same goes for foo.mount.d/, of course you can't really control things like "After=" or "Options=" for a mount that already happened, but you can add a [Unit] Wants= through drop-ins; my autofs-managed mounts seem to consistently trigger the test service I've added in both ways.

By Etienne Dechamps at 2021-10-17 12:18:12:

Maybe I misunderstand the problem you're trying to solve, but perhaps zfs-mount-generator(8) could alleviate at least some of these problems? Especially the custom properties?

Would systemd.path(5) be of some use?

Defines paths to monitor for certain changes: PathExists= may be used to watch the mere existence of a file or directory. […] DirectoryNotEmpty= may be used to watch a directory and activate the configured unit whenever it contains at least one file.

Units also have various ConditionPath* checks:

ConditionDirectoryNotEmpty= is similar to ConditionPathExists= but verifies that a certain path exists and is a non-empty directory.

By cks at 2021-10-19 10:53:26:

Automounting adds an extra layer of undesired indirection over top of a bind mount, and it's not clear from the systemd documentation what happens if the .mount unit fails when the automount unit tries to activate it. At best this is "on demand" bind mount attempts, which may eventually succeed, instead of a bind mount triggered (reliably) when the underlying mount appears.

If I'm understanding the idea right, a WantedBy (or Wants) doesn't deal with things being asynchronous, as far as I know and can see from the systemd documentation. The bind mount Wants the underlying real mount (whether this is expressed by itself or by the real mount having a WantedBy of the bind mount), but this merely means that the bind mount will fail if the real mount isn't there when it tries to start; it doesn't activate the bind mount when the real mount appears.

Inverting the Wants relationship so that the bind mount says it is WantedBy the underlying real mount doesn't work. If the real mount isn't there, the bind mount still tries to happen (since it's only a Wants). You don't want to use a RequiredBy instead, because then if the bind mount fails for some reason the real mount will at least be considered to have failed, which is not right and almost certainly not wanted.

(If you put a Condition on the bind mount that isn't satisfied if the real mount doesn't exist, systemd v237 doesn't retry the mount later; it sticks at 'start condition failed'.)

My guess is that zfs-mount-generator won't help, because the fundamental issue is units that don't activate immediately but that we want to trigger other units. Having real .mount units for ZFS filesystems probably doesn't help if the ZFS filesystems don't appear promptly when systemd is starting everything.

(This comment reply has been delayed by me wanting to find time to test some things brought up here.)

By cks at 2021-10-19 11:20:09:

On systemd path units, testing says that they don't notice when a mount appears that has what they're looking for. If you have a path unit with PathExists=/cs/mail/.flag and it starts before the /cs/mail mount, it won't notice when the mount happens and the flag file is now present. This is probably an intrinsic limit in the underlying kernel inotify mechanism that path units use.

(If you 'systemd restart <what>.path', it does notice and will trigger your bind mount, but there's nothing to auto-restart your path units.)

From at 2021-10-22 09:20:07:

If I'm understanding the idea right, a WantedBy (or Wants) doesn't deal with things being asynchronous, as far as I know and can see from the systemd documentation. The bind mount Wants the underlying real mount (whether this is expressed by itself or by the real mount having a WantedBy of the bind mount), but this merely means that the bind mount will fail if the real mount isn't there when it tries to start; it doesn't activate the bind mount when the real mount appears.

The point is that you could have the real mount Want the bind mount. It's a dependency loop, but not necessarily an ordering loop: the bind mount would have Wants/After=real, but the real mount would have Wants/Before=bindmount.

By cks at 2021-10-22 13:50:34:

It looks like I messed up in my first round of testing a bind mount that set a WantedBy of the underlying mount, because it now works for me on the Ubuntu 18.04 LTS systemd v237. However, at least in this environment it has the problem that if I unmount the bind mount, systemd goes ahead and unmounts the underlying mount too (which then gets ZFS a bit upset). This is definitely not what I want in a general triggering relationship; I should be able to shut down the triggered thing (here, the bind mount) without the triggering thing getting shut down by systemd.

(I think I probably left my bind mount also WantedBy, so that systemd attempted to activate it on boot even when the underlying mount wasn't there because that's what I told it to do.)

By foobar at 2022-01-27 00:08:33:

You can get a lot of the way there with a ExecStartPost= directive, a ExecStartPre= directive, and a .path service file.

Have the triggering service (foo.service) create a temporary file via a ExecStartPost= directive (touch /path/to/some/file).
Have the triggered service (bar.service) have a corresponding .path unit file (bar.path) which is watching for the creation of the file created by the triggering service (ie. bar.path watches for foo.service to create /path/to/some/file).
Have the triggered service (bar.service) have a ExecStartPre= (or maybe ExecStartPost=) directive to clean up the file.

You could also just have a systemctl start call in the ExecStartPost= directive instead, but then you lose the declarative nature of systemd.

Oh and you probably can clean this up by abusing the JoinsNamespaceOf= directive to have the services share a file system namespace.

Written on 16 October 2021.
« You may not want to require all of your bind mounts (in systemd)
Getting some hardware acceleration for video in Firefox 93 on my Linux home desktop »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Oct 16 23:27:07 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.