Systemd needs (or could use) a linter for unit files

March 4, 2021

Today I made a discovery:

Today's Fedora packaging failure: /usr/lib/systemd/system/lttng-session.service (from lttng-tools) is a PGP key, not a systemd .service unit. (Insert joke about people once again not managing to use PGP properly)

Yes, bug filed: #1935426

I discovered this because I was watching 'journalctl -f' while I upgraded my office workstation to Fedora 33 in my usual way. The upgrade process causes systemd to re-examine your unit files and complain about things. Most of the complaints were normal ones like:

mcelog.service:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether
xinetd.service:10: PIDFile= references a path below legacy directory /var/run/, updating /var/run/ → /run/; please update the unit file accordingly.

But mixed in with those complaints I noticed the much more unusual:

lttng-sessiond.service:1: Assignment outside of section. Ignoring.
lttng-sessiond.service:3: Assignment outside of section. Ignoring.
[.. for a bunch more lines ..]

That made me curious what was in the file, whereupon I discovered that it was actually a PGP public key instead of a systemd unit file.

We can laugh at this mistake because it's funny in several different ways (given that it involves PGP for extra spice). But it's actually pointing out a systematic problem, one that is also illustrated by those other messages about other systemd units, which is that there's no easy way to check your unit files to see if systemd is happy with them. In other words, there is no linter for systemd unit files.

If there was a linter, none of these problems would be there, or at least any that were still present would be ones that Fedora (or any other Linux distribution) had decided were actually okay. With a linter, Linux distributions could make it a standard packaging rule (in whatever packaging system they use) that all systemd units in a package had to pass the linter; this would have automatically detected the lttng-tools problem, probably among others. Without a linter, the only way to detect systemd unit problems is to enable them and see if systemd complains. This is not something that's easy to automate, especially during package builds, and so it's fallible and limited.

(Because of that it invites people to file bugs for things that may not be bugs. Are these issues with the PIDFile location actual oversights in the packaging or an area where Fedora's standard doesn't line up with the systemd upstream? I can't tell.)

An automatically applied linter would be especially useful for the less frequently used packages and programs, where an issue has a much easier time lurking for some time. Probably not very many people have lttng-tools even installed on Fedora, and clearly not very many of them use things that require the lttng sessiond service.

PS: This isn't the only systemd project where standards have changed and some systemd bit is now complaining. Systemd-tmpfiles complains about various things wanting to clean up bits in /var/run, for example.

Comments on this page:

By Tom Wassenberg at 2021-03-05 06:15:23:

systemd-analyze verify FILE... can be used to verify the correctness of systemd unit files. It checks for unknown sections/directives and a few other things, so it should have caught the example here.

There is a linter! It's systemd-analyze verify.

I haven't tested what it does with a completely invalid unit file, but it has been helpful for me before for identifying problems like misspelled directives.

One thing it is missing (as far as I know!) is a way to run it against all installed units. I've had to resort to shell pipelines for that.

By cks at 2021-03-05 09:39:52:

I didn't know about systemd-analyze, so it's good to find out about it. But at least for me it doesn't quite behave the way you want a linter (although it does behave well for what it is). On Fedora 33 with systemd 246, it also complains about all issues in enabled services. In practice on Fedora 33 this drowns any specific issues with a particular unit file with complaints about other ones.

(It does properly detect something that's not even a systemd unit file, with complaints that looked basically the same as for the PGP public key.)

By jbowen at 2021-03-05 11:21:14:

I wish I would have known about systemd-analyze verify. I once spent a day trying to figure out why a unit file wasn't working (no complaints in dmesg or the journal), only to find I'd transposed two letters in a directive.

By mjeanson at 2021-03-05 15:38:44:

My bad, I'm not a fan of rpm spec files, the reordering of source files in the header metadata ended with the wrong file being installed as a systemd unit.

This systemd service is a convenience for people that do kernel tracing which ins't possible in a bare fedora install, as they don't allow out-of-tree kernel modules to be packaged. That might explain why it wasn't caught earlier.


By cks at 2021-03-05 16:08:45:

In case it wasn't clear in the entry, I think blaming a person for this sort of issue is asking far too much of package maintainers. Mistakes and omissions will be made, so automation should be there to catch them.

(This is the case in any situation, but it's especially so for packaging in most Linux distributions since there are a lot of packages and not all that many package maintainers, and it's grinding repetitive work, which is exactly the sort of work people do very badly at.)

From at 2021-03-05 18:47:31:

No hard feelings, I just thought I'd follow up on the bug. I agree about automation and tooling being the right answer to these kind of problems.

I package lttng for multiple distro and I can say Fedora has the best and most streamline tooling for packaging, yet it could still be so much better.

Written on 04 March 2021.
« What signal a RJ-45 serial connection is (probably) missing
How not to use Apache's RewriteRule directive in a reverse proxy »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Mar 4 23:53:08 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.