I wish that systemd (and everything) would rate-limit configuration warnings

August 2, 2022

If you have an Ubuntu 22.04 machine and you use the 'oidentd' package, you're probably seeing a lot of noise in your logs that looks like this:

systemd[1]: /lib/systemd/system/oidentd@.service:10: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.

The obvious reason you're seeing this message is that the 22.04 'oidentd@.service' template service unit specifies the now-obsolete 'StandardError=syslog' setting. You're seeing this message frequently because systemd apparently generates these messages whenever the unit is started and oidentd.socket is set to start it on every connection (ie, oidentd.socket specifies 'Accept=yes', which is also the only time you can have a templated socket service unit).

The not so obvious reason you're seeing this message frequently is that systemd, like many other things, has not put ratelimits on warnings about potential configuration issues. I'm sure that the systemd people thought that warning each time the unit was started was harmless; after all, units are only started infrequently, right? Well, no, not always. As we can see here, it's entirely possible for some service units to be started a lot.

As I mentioned, systemd is not the only thing that's had this issue. For instance, the Prometheus host agent supports a bunch of 'collectors' and has metrics pulled from it frequently by the Prometheus server. Every so often one of the collectors that are enabled by default decides that it should emit a warning to the effect of 'this system doesn't support me' every time Prometheus pulls metrics.

(Since the default pull interval is 15 seconds, these messages add up very fast.)

My general view is that the ideal rate limit for this sort of warning about configurations is 'once', until the runtime configuration changes. For systemd, report a warning the first time oidentd@.service is used as a template unit, and then don't do it again until at least a 'systemctl daemon-reload' tells systemd to reload the unit configurations (even better would be 'until the oidentd@.service unit file changes', but that might be harder).

Written on 02 August 2022.
« A brief history of looking up host addresses in Unix
Vim settings I'm using for editing YAML (with a sideline into Python) »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Aug 2 22:47:30 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.