Using systemd timers to run things frequently (some early notes)

August 26, 2022

If you're satisfied with running something no more often than once a minute, /etc/cron.d entries are the easiest approach and what I use. But today I wound up wanting to run something more frequently. While there are various ways to do this with various degrees of hackery, it seemed like a good time to try out systemd timer units.

First we will need our .service unit, which is straightforward:

[Unit]
Description=Read sensors

[Service]
Type=oneshot
# Run your script
ExecStart=/opt/cslab/generate-sensor-metrics

The skeleton of the corresponding .timer unit is:

[Unit]
Description=Timer for reading sensors

[Install]
WantedBy=timers.target

[Timer]
AccuracySec=5s
... we need some magic here ...

(Setting AccuracySec= insures that systemd runs our service more or less when we ask it to, instead of delaying. You may want to be narrower here, for example saying '1s' as the required accuracy.)

The remaining thing we need is something to tell systemd that, for example, we want this to run every 30 seconds. There are at least two options. First, we can use OnCalendar= to set a systemd time specification to run, say, on :05 and :35 of every minute:

OnCalendar=*:*:05,35

(Another option for how to specify this sort of time is '*:*:0/30', which means more or less what it would in cron.)

This is much like cron time specifications except it allows us to go down to seconds instead of stopping at minutes the way that cron does. After looking at things and writing this entry, I've come to feel that using OnCalendar is your best and simplest option if the time intervals you want divide evenly into a minute (eg things like 'every 30 seconds', 'every 15 seconds', or 'every 10 seconds') and you're not allergic to always running your script at specific seconds.

Alternately we can try to say 'activate this every 30 seconds', or whatever number of seconds you want (including numbers that deliberately don't evenly divide into minutes, such as 31 seconds). Systemd doesn't have a straightforward way of expressing this that I can see; instead, you get to say 'do this 30 seconds after <some events>', and then you pick the events. You need at least two events; one to start things initially, when the system boots or you start the timer, and one to make it repeat. I currently feel that the best single option for a starting event is OnActiveSec=, because that covers all of the various cases where the timer gets activated, not just when the system boots. If you stop the timer for a while and then do 'systemctl start <thing>.timer', this will kick it off.

There are two options for getting things to repeat, OnUnitActiveSec and OnUnitInactiveSec, which define delays from when the unit started and when the unit stopped respectively. I believe that if you want there to be a 30 second delay between successive runs of your script, then you want to use OnUnitInactiveSec. On the flipside, if you want things to tick every 30 seconds no matter how long the script took, you want OnUnitActiveSec.

So a plausible [Timer] section is:

[Timer]
AccuracySec=5s
OnActiveSec=30s
OnUnitActiveSec=30s

You could set 'OnActiveSec' to a smaller value, since all it is there for is to trigger the initial service unit activation that then starts the every 30 second ticker. Generally, the timer will be activated as the system boots, so you'll start ticking 30 seconds after that.

In all cases, I believe that using a systemd timer and service unit means that only one copy of your script will ever be running at one time. Timer units have the effect of activating your service unit, so if your service unit is already (or still) active, systemd does nothing rather than activating a second copy.

One of the downsides of using systemd timers for this is that you get a certain amount of log spam from it. Every time the timer unit starts your service unit, you'll most likely get three log lines:

Starting Read sensors
<thing>.service: Deactivated successfully.
Finished Read sensors

Using crontab generally gets you only one log line per invocation. On the other hand, your (systemd) logs may already be getting flooded from other things; this is definitely the case on some of our machines.

PS: One reason to pick non-divisible numbers of seconds, such as 31, is to insure that you never synchronize with something else that happens either on fixed seconds or at some other fixed interval, like 'every 15 seconds'. However, see the RandomizedDelaySec= and FixedRandomDelay= timer unit settings for other potential options here.


Comments on this page:

The 31s and 15s job timers will still coincide every 465s. By picking coprime numbers, as you have, you ensure that the repeat will never be shorter than the product of the two, but you can't avoid the coincidence coming round in its time. Random timers at least avoids predictable repetition, and coincidence becomes a matter of probability, but it will still happen, on average at the same frequency. AccuracySec blurs the opportunities for coincidence, but doesn't make them less likely.

Sadly, if you want to be absolutely sure two jobs never run at the same time, a little more locking and checking is required.

By Carl at 2022-08-27 12:07:21:

Cicada timing unit.

By Juan at 2022-09-29 19:18:08:

One thing I found extremely helpful was using multiple directives to construct a timer that occurs periodically over night. For example, the timer below runs a transcode script every 5 minutes between 8PM and 8AM with a 120s randomized delay between triggers.

#Transcode.timer
[Unit]
Description=Transcode new files by checking every 5 minutes between 8PM and 8AM

[Timer]
OnCalendar=00..08:0/5
OnCalendar=20..23:0/5
OnBootSec=120
RandomizedDelaySec=120

[Install]
WantedBy=timers.target

I use this to transcode files that get pushed to and/or collected by my server through several different methods (eg. sftp, ssh, HTTP upload, S3 copy). The transcode service also uses IOSchedulingClass=idle to de-prioritize the transcode compared to the various methods saving the files.

As for monotonic (OnUnit*) vs realtime (OnCalendar) timers, I use both. Predominantly I use the OnCalendar directive, but when I do spring for monotonic timers, I always grab OnUnitInactive coupled with an OnBootSec.

Written on 26 August 2022.
« U.2, U.3, and other server NVMe drive connector types (in mid 2022)
Large scale Internet SSH brute force attacks seem to have stopped here »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Fri Aug 26 22:52:29 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.