We're broadly switching to synchronizing time with systemd's timesyncd

November 30, 2017

Every so often, simply writing an entry causes me to take a closer look at something I hadn't paid much attention to before. I recently wrote a series of entries on my switch from ntpd to chrony on my desktops and why we don't run NTP daemons but instead synchronize time through a cron entry. Our hourly crontab script for time synchronization dates back to at least 2008 and perhaps as early as 2006 and our first Ubuntu 6.06 installs; we've been carrying it forward ever since without thinking about it very much. In particular, we carried it forward into our standard 16.04 installs. When we did this, we didn't really pay attention to the fact that 16.04 is different here, because 16.04 is systemd based and includes systemd's timesyncd time synchronization system. Ubuntu installed and activated systemd-timesyncd (with a stock setup that got time from ntp.ubuntu.com), we installed our hourly crontab script, and nothing exploded so we didn't really pay attention to any of this.

When I wrote my entries, they caused me to start actually noticing systemd-timesyncd and paying some attention to it, which included noticing that it was actually running and synchronizing the time on our servers (which kind of invalidates my casual claim here that our servers were typically less than a millisecond out in an hour, since that was based on ntpdate's reports and I was assuming that there was no other time synchronization going on). Coincidentally, one of my co-workers had also had timesyncd come to his attention recently for reasons outside of the scope of this entry. With timesyncd temporarily in our awareness, my co-workers and I talked over the whole issue and decided that doing time synchronization the official 16.04 systemd way made the most sense.

(Part of it is that we're likely to run into this issue on all future Linuxes we deal with, because systemd is everywhere. CentOS 7 appears to be just a bit too old to have timesyncd, but a future CentOS 8 very likely will, and of course Ubuntu 18.04 will and so on. We could fight city hall, but at a certain point it's less effort to go with the flow.)

In other words, we're switching over to officially using systemd-timesyncd. We were passively using it before without really realizing it since we didn't disable timesyncd, but now we're actively configuring it to use our time local servers instead of Ubuntu's and we're disabling and removing our hourly cron job. I guess we're now running NTP daemons on all our servers after all; not because we need them for any of the reasons I listed, but just because it's the easiest way.

(At the moment we're also using /etc/default/ntpdate (from the Ubuntu ntpdate package) to force an initial synchronization at boot time, or technically when the interface comes up. We'll probably keep doing this unless timesyncd picks up good explicit support for initially force-setting the system time; when our machines boot and get on the network, we want them to immediately jump their time to whatever we currently think it is.)


Comments on this page:

By anonymous at 2017-12-01 07:54:58:

If you're feeling evil, forwarding all ntp traffic hitting router to internal ntp server can save time configuring the clients.

From 78.60.211.195 at 2017-12-01 09:10:22:

I guess we're now running NTP daemons on all our servers after all

timesyncd only does SNTP (one source), so if you're worried about code complexity, it's not too far from ntpdate.

We'll probably keep doing this unless timesyncd picks up good explicit support for initially force-setting the system time;

I think that should already work -- it doesn't seem like the spike detection could possibly trigger during the first sync... Regardless, explicit support shouldn't be difficult to implement, since it already performs a resync whenever the system clock is changed, and if systemd-networkd is in use, whenever the network state changes.

Though, I would expect server RTCs to remain working even when powered off? So they shouldn't need a large jump in any case, should they?

By Dan.Astoorian at 2017-12-01 11:59:30:

Though, I would expect server RTCs to remain working even when powered off? So they shouldn't need a large jump in any case, should they?

Raspberry Pis, for one notable example, have no RTC.

Even if the hardware has one, there are many interesting ways things can go wrong with them (failed CMOS battery or whatever the modern-day equivalent of that is, DST changes if the clock is not stored in UTC, etc.).

From 78.60.211.195 at 2017-12-02 08:18:31:

Raspberry Pis, for one notable example, have no RTC.

I don't think cks runs many services on Raspberry Pis...

Though systemd-timesyncd has a related feature where it stores time on disk (every now and then, also during shutdown), and whenever you boot, systemd will forward the clock to that time.

DST changes if the clock is not stored in UTC,

Usually not a concern unless the servers are Windows-based.

At the moment we're also using /etc/default/ntpdate (from the Ubuntu ntpdate package) to force an initial synchronization at boot time, or technically when the interface comes up. We'll probably keep doing this unless timesyncd picks up good explicit support for initially force-setting the system time; when our machines boot and get on the network, we want them to immediately jump their time to whatever we currently think it is.

While I don't use it, I thought this was the whole point of timesyncd. What does it do on boot for you? Does it slew the time, as opposed to step it?

Written on 30 November 2017.
« The cost of memory access across a NUMA machine can (probably) matter
I'm basically giving up on syslog priorities »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Nov 30 21:37:12 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.