Formatting alert start and end times in Prometheus Alertmanager messages

June 4, 2020

One of the things that you'll generally wind up doing with Prometheus's Alertmanager is creating custom alert messages (in whatever medium you send alerts, whether that's email or something else). Alertmanager comes with some default templates for various alerting destinations, but they're very generic and not all that useful in many situations. If you're customizing alert messages, one thing you might want to put in is when the alerts started (and possibly when they ended, if you send messages when alerts are 'resolved', ie stop). For instance, you might want a part of the message that looks like this:

  • some text about the alert
    (alert active from 17:52:49 2018-10-21 to 17:58:19 2018-10-21)

In Prometheus alerts, there are two places you can format things, with somewhat different features available in them; you can format data through templating in alert annotations, or through templating in the actual Alertmanager message templates. Both of these use Go templating, but they have somewhat different sets of formatting functions available (generally alert annotations have a richer set of formatting functions).

The start time and possibly the end time of alerts are available in Alertmanager templating as StartsAt and EndsAt names on each alert object (which are in turn accessed through, eg, Alerts). These are Go time.Time values, and so are formatted through their .Format() method. You do this formatting like the following (the range is for illustrative purposes):

{{ range .Alerts.Firing }}
   alert started at {{ .StartsAt.Format "15:04:05 2006-01-02" }}
{{ end }}

There are two things wrong with this formatting example, one of which is visible in the message about. First, it doesn't tell you what time zone the formatted time is in. Second, it doesn't force this time zone to be anything in specific, which matters because Go time.Time values have time zones associated with them that change how a given absolute time is presented. Since we're not forcing a time zone and not displaying the StartsAt's time zone in the formatted time, we have no idea what time this really is. In my example message above, it's '17:52:49' in some time zone, but we have no idea which one.

To deal with both issues, the correct way in Alertmanager templates to format a time is:

{{ .StartsAt.Local.Format "15:04:05 MST 2006-01-02" }}

This will give you a formatted time like '22:16:02 EDT 2020-06-04', which is clear and explicit (if you want, you can be more explicit in various ways; see the Go time formatting documentation.

(Change the .Local to .UTC if you want your alert times in UTC time instead of Alertmanager's local time zone, whatever you've set that to.)

Incidentally, we were previously using the first form, and we had our formatted alert time stamps in our alert email silently change from local time to UTC after we upgraded Prometheus (not Alertmanager) from 2.17.2 to 2.18.1 back in early May. This is because of this commit for issue 7066, which was considered so small that it didn't even appear in the Prometheus release notes. Without the time zone explicitly named in our messages, it took some time before an alert co-worker noticed that the times looked odd (and working out what was going on involved some head scratching).

Written on 04 June 2020.
« In theory you (we) should have SPF records for HELO hostnames too
Why we put alert start and end times in our Prometheus alert messages »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Jun 4 23:34:10 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.