Putting a footer on automated email that says what generated it

November 10, 2019

We have various cron jobs and other systems that occasionally send us email, such as notifications about Certbot renewals failing or alerts that Prometheus is down. One of our local system administration habits when we set up such automated email is that we always add a footer to the email message that includes as much information as possible about what generated the message and where.

For example, the email we send on systemd unit failures (currently used for our Certbot renewal failures) has a footer that looks like this:

(This email comes from /opt/cslab/sbin/systemd-email on $HOST. It was probably run from /etc/systemd/system/cslab-status-email@.service, triggered by an 'OnFailure=' property on the systemd unit $SERVICE.)

I acquired this habit from my co-workers (who were doing it before I got here), but in an environment with a lot of things that may send email once in a while from a lot of machines, it's a good habit to get into. In particular, if you receive a mysterious email about something, being able to go find and look at whatever is generating it is very useful for sorting out what it's really telling you, what you should do about it, and whether or not you actually care.

(There is apparently local history behind our habit of doing this, as there often is with these things. Many potentially odd habits of any particular system administration environment are often born from past issues, much as is true for monitoring system alerts. In both cases, sometimes this is bolting the stable door after the horse is long gone and not returning and sometimes it's quite useful.)

In general, this is another way of adding more context to things you may only see once in a while, much like making infrequent error messages verbose and clear. We're probably going to have forgotten about the existence of many of our automated scripts by the time they send us email, so they need to remind us about a lot more context than I might think when writing them.

PS: If you're putting a footer on the automatic email, you don't necessarily have to identify the machine by the GECOS name of its root account, but every little bit helps. And identifying the machine in the GECOS name still helps for things that don't naturally generate a footer, like email from cron about jobs that failed or produced unexpected output.

Written on 10 November 2019.
« The problems with piping curl to a shell are system management ones
An apparent hazard of removing Linux software RAID mirror devices »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Nov 10 23:30:20 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.