>PS: I think that you can arrange to get emailed if certbot.service fails, by using a drop in to add an 'OnFailure=' that starts a unit that sends email when triggered. But I don't think there's a good way to dig the actual error messages from the most recent attempt to start the service out of the journal, so the email would just be 'certbot.service failed on this host, please come look at the logs to see why'.

This is already documented [1] and this post is spreading misinformation. Notice how the group runs as systemd-journal, so you can access the logs of the last failed attempt.

[1] https://wiki.archlinux.org/index.php/Systemd/Timers#MAILTO

keur: that's cute how you need to know systemd inside out to think up how to plug in your separate script for sending an e-mail on error (because OnFailure is not documented in systemd.timer(5)), a feature that cron has built in.

OnFailure is of course documented in systemd.unit(5) because it applies to all kinds of units. (Although, if it's not defined in systemd.timer(5), I would not be sure what a "failed timer" means. Maybe the timer is successful if the service is found, regardless of the service's status.)

We collect logs through a central service and have an alert set up to at least capture any "systemd service failed" alerts. We still have to go look at the log aggregator to find more details. Of course, sending logs to a third party doesn't work for everyone.

It's yet another reason to hate systemd. You always find out yet another workflow has been Changed For Your Own Good™ and have to track down all-new commands to deal with it in the middle of a crisis. And the things you can put in unit files aren't even stable across versions, so what you fix today might be broken tomorrow.

dozzie: It doesn't require knowing systemd inside and out because it is on the Arch Wiki. You are just bad.

My own view is that while the Arch Wiki is a fine resource, it is not the same thing as systemd's actual documentation, which is in manpages installed on your system and on systemd's official site. The actual, official documentation provides neither warning nor fixes on this issue. It is nice that people have documented workarounds and fixes for this; it is not nice that they are not in the official documentation or that the out of the box state is dangerous, different from cron, and cannot be fixed inside systemd without using extra scripts.

If systemd timers are supposed to be a drop in replacement for cron.d entries, they should be as easy to operate and work as well, and do so without needing you to read third party wiki pages and assemble local scripts and local service units. Other systemd replacements for things have generally had this level of easy operation.

sapphirepaw: for my purposes, I guess 'failed timer' means both a failed .timer unit (if that can even happen) and a failed .service unit for the .timer unit. Here, what failed is the Certbot service unit; the Certbot timer worked fine (in that it triggered the service unit).

cks: Yes I agree it should probably be included in the offical documentation. I think you a reaching when calling systemd timers a "drop in replacement" for cron. It doesn't say that anywhere on the man page. Even the OnCalendar isn't 1:1 with cron syntax. It is an alternative to cron, not a replacement.

One of the social problems on display in Certbot's case is that developers and software packagers are evidently being led to switch from cron.d jobs to systemd timers. The Ubuntu PPA Certbot packaging ships both a cron.d job and a systemd timer unit, but it prefers the systemd timer unit (the cron job disables itself if it detects that systemd is running); the Fedora Certbot package doesn't even have a cron.d file (and it doesn't arrange to send email on failure). There is at least an implicit view floating around that systemd timer units are the modern drop in replacement for cron.d jobs and you should switch from the latter to the former. As exhibited here, they are not equivalent and there are implications for making the switch, implications which may not be clearly understood in at least some cases.

(Various non-systemd documentation about timers talks about them as a cron replacement, too, although I don't think any systemd documentation does.)

All I meant about "failed timer" was that you can apparently put OnFailure into a *.timer file, but it's not obvious what that means to systemd (on what condition would that actually run?) If you're not even supposed to put that directive in the timer file at all, apparently that's another point of confusion systemd has introduced here. IDK.

And other funny thing that the timers have is that you can't restart an active unit, like for example, fire a nightly restart to the foo service. Systemd timer "simply left running".

It's documented in systemd.timer, but why the narrow the .timer functionality?, what they gain with it?

Switching from cron to systemd timers is definitely an operational change.

The emphasis on emails feels like status quo bias, though. Imagine the situation was reversed: that everything was using systemd timers and then someone wrote cron and people started switching to that. In that case, there is a similar operational change. You'd switch from having a centralized status (e.g. systemctl list-units --failed) and centralized logging (the journal, which also defaults to forwarding to syslog) to crond sending emails. Is that an improvement, a step backwards, neither or both? Either way, I'd say the most important thing is that you need to integrate the new tool into your environment.

FWIW, at my work, we are in the process of converting all of our cron jobs into systemd service and timer pairs. One of the big reasons for that is that we already have systemd failed units monitored by Icinga, so this eliminates a separate way of monitoring things (emails to root) in favor of our unified alarming system. Also, emails are not great if an "every minute" or "every five minute" cron job starts failing.

We also expect other advantages. For one, service units are easier to develop & debug, as you can just start them with systemctl, without having to fiddle with the cron definition to run it at "the next minute" and then remember to change it back to the production timings when you're done. Also, systemd timers can be randomized to spread the load rather than having every system wake up at the same moment and start running e.g. daily jobs. (I'm aware that cronie has RANDOM_DELAY, but Debian & Ubuntu still use Debian's vixie-cron which does not.)

Time will tell if this was a good idea or not. Assuming this goes well for us, the next phase will be to switch from crond to systemd-cron, a (third-party) systemd generator that creates service and timer units from crontabs. This will dynamically convert any package cron jobs.

If emails are what you want, systemd timers are definitely a step backwards in that regard. Emails can be done, and systemd-cron has a setup for them for the units it converts, but it is additional work. And for timer-triggered services that are provided by distro packages (i.e. not you), while you can use drop-in config files to add the relevant configuration, you have to do that per-service. This is extra work, and more importantly, you have to know about all such units you have installed, which does not scale.

Another, more general, option would be to wire up something to check for failed units and send an email based on that.

Hello, We had the exact same issue with certbot. I came up with another workaround. See this question https://superuser.com/questions/1621517/having-all-systemd-timers-mail-to-root-on-error/1699235#1699235

And some opinions.

It is another case of classic systemd disrespect for linux traditions. I think that there should be a way to fallback to old cron's behavior. On the other hand cron's attitude might not be the most reasonable for some people. And again it would be better if systemd provided the configuration options for admins to opt in to mail output.This would ease the operational change (to move from cron to systemd.timers).