== Formatting alert start and end times in Prometheus Alertmanager messages One of the things that you'll generally wind up doing with [[Prometheus https://prometheus.io/]]'s [[Alertmanager https://prometheus.io/docs/alerting/alertmanager/]] is creating custom alert messages (in whatever medium you send alerts, whether that's email or something else). Alertmanager comes with some default templates for various alerting destinations, but they're very generic and not all that useful in many situations. If you're customizing alert messages, one thing you might want to put in is when the alerts started (and possibly when they ended, if you send messages when alerts are 'resolved', ie stop). For instance, you might want a part of the message that looks like this: > * some text about the alert \\ > (alert active from 17:52:49 2018-10-21 to 17:58:19 2018-10-21) In Prometheus alerts, there are two places you can format things, with somewhat different features available in them; you can format data through templating in [[alert annotations https://prometheus.io/docs/prometheus/latest/configuration/template_examples/]], or through templating in [[the actual Alertmanager message templates https://prometheus.io/docs/alerting/notifications/]]. Both of these use [[Go templating https://golang.org/pkg/text/template/]], but they have somewhat different sets of formatting functions available (generally [[alert annotations have a richer set of formatting functions https://prometheus.io/docs/prometheus/latest/configuration/template_reference/]]). The start time and possibly the end time of alerts are available in Alertmanager templating as _StartsAt_ and _EndsAt_ names on each alert object (which are in turn accessed through, eg, _Alerts_). These are Go [[_time.Time_ https://golang.org/pkg/time/#Time]] values, and so are formatted through their [[_.Format()_ https://golang.org/pkg/time/#Time.Format]] method. You do this formatting like the following (the _range_ is for illustrative purposes): .pn prewrap on > {{ range .Alerts.Firing }} > alert started at {{ .StartsAt.Format "15:04:05 2006-01-02" }} > {{ end }} There are two things wrong with this formatting example, one of which is visible in the message about. First, it doesn't tell you what time zone the formatted time is in. Second, it doesn't force this time zone to be anything in specific, which matters because [[Go time.Time values have time zones associated with them ../programming/GoTimeHasLocation]] that change how a given absolute time is presented. Since we're not forcing a time zone and not displaying the _StartsAt_'s time zone in the formatted time, we have no idea what time this really is. In my example message above, it's '17:52:49' in some time zone, but we have no idea which one. To deal with both issues, the correct way in Alertmanager templates to format a time is: > {{ .StartsAt.Local.Format "15:04:05 MST 2006-01-02" }} This will give you a formatted time like '22:16:02 EDT 2020-06-04', which is clear and explicit (if you want, you can be more explicit in various ways; see [[the Go time formatting documentation https://golang.org/pkg/time/#pkg-constants]]. (Change the _.Local_ to _.UTC_ if you want your alert times in UTC time instead of Alertmanager's local time zone, whatever you've set that to.) Incidentally, we were previously using the first form, and we had our formatted alert time stamps in our alert email silently change from local time to UTC after we upgraded Prometheus (not Alertmanager) from 2.17.2 to 2.18.1 back in early May. This is because of [[this commit for issue 7066 https://github.com/prometheus/prometheus/commit/7646cbca328278585be15fa615e22f2a50b47d06]], which was considered so small that it didn't even appear in the Prometheus release notes. Without the time zone explicitly named in our messages, it took some time before an alert co-worker noticed that the times looked odd (and working out what was going on involved some head scratching).