Systemd unit templates don't provide a native way to have multiple parameters

December 19, 2022

We have a collection of Apache servers, and as part of our Prometheus metrics system we indirectly scrape metrics from them, using a third party Apache exporter that queries Apache's mod_status information and converts it to Prometheus metrics. Because the exporter's author chose to write it that way, you need to run one instance of the exporter per Apache server you want to extract metrics from. Ordinary people would probably run these exporter instances on the Apache hosts themselves. We opted to instead run all of the exporters on our metrics server, which means that we need one systemd unit per exporter instance. As we gather metrics from more and more Apache servers, this had led to more and more systemd units, each of them basically identical.

On the surface this sounds like a great use for a systemd template unit file. We could create a templated 'apache-exp@.service' file, and then start 'apache-exp@server1', 'apache-exp@server2', and so on. However, there is a problem that I haven't yet mentioned: because they're all running on the same host, each of these exporter instances needs to be told not just what Apache to query but also what local port to listen on (since they can't all listen on the same one, the way they could if they were on separate servers).

One obvious way of providing these two arguments to a template unit is some format of multi-part instance names, for example 'server1:9112'. Systemd will create multi-part instance names for templated socket units, but there are no systemd specifiers that will separate an instance name into parts for you. Any separation would have to be done by a front end script, taking the '%I' (or '%i') from systemd and breaking it up into the expected parts. This is perfectly doable but does mean adding a local script, with all that entails.

Systemd can do some processing of command lines, but it doesn't support Bourne shell style extraction of parts of variables. In theory your systemd ExecStart can be a 'sh -c' invocation (there's an example in the Command lines section), but in practice this feels like a lot of (fragile) work just to get some inline extraction of two parts of the instance name.

(You'll also have to put the instance name in a shell variable, because you can only use '${...%:}' and '${...#:}' with variables, not with literal text that you shove in. This is a fair Bourne shell limitation.)

I think this is a fair limitation for systemd instance name handling to have. The situation I'm in is somewhat obscure and handling anything like it would get complicated fast. Systemd is not the place to put powerful text rewriting and substitution mangling, because honestly we've seen where that goes and it often doesn't go anywhere good once you start.


Comments on this page:

By dkess at 2022-12-19 22:35:35:

systemd also has a line length limit (which applies even if you break up a line with the backslash syntax). If you're writing a long complex shell invocation in your ExecStart line it will break the unit if it's too long. Much better to use a secript.

One solution that I found to this problem is using environment files like this:

[Service]
...
EnvironmentFile = /etc/kawipiko/%i.env
ExecStart = /usr/local/bin/kawipiko-server --bind ${KAWIPIKO_BIND_IP}:${KAWIPIKO_BIND_PORT} --archive ${KAWIPIKO_ARCHIVE}

Here the some-site.env file contains simple NAME = value pairs, that can then be expanded by systemd like for example KAWIPIKO_BIND_IP. This expansion can be applied to many more attributes than just ExecStart, like for example user, limits, etc.

Although I don't like the solution too much, I think it's quite in line with systemd's policy of having many files (the unit file, the environment file, the socket file, etc.) to configure one single service...

By Etienne Dechamps at 2022-12-20 13:55:55:

Systemd issue #14895 is a request to support multiple instance parameters. Sadly it didn't go anywhere.

Written on 19 December 2022.
« My dmenu wrapper script and what it will invoke for me
Detecting missing or bad Go modules and module versions »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Dec 19 22:21:50 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.