Wandering Thoughts archives

Putting a footer on automated email that says what generated it

We have various cron jobs and other systems that occasionally send us email, such as notifications about Certbot renewals failing or alerts that Prometheus is down. One of our local system administration habits when we set up such automated email is that we always add a footer to the email message that includes as much information as possible about what generated the message and where.

For example, the email we send on systemd unit failures (currently used for our Certbot renewal failures) has a footer that looks like this:

(This email comes from /opt/cslab/sbin/systemd-email on $HOST. It was probably run from /etc/systemd/system/cslab-status-email@.service, triggered by an 'OnFailure=' property on the systemd unit $SERVICE.)

I acquired this habit from my co-workers (who were doing it before I got here), but in an environment with a lot of things that may send email once in a while from a lot of machines, it's a good habit to get into. In particular, if you receive a mysterious email about something, being able to go find and look at whatever is generating it is very useful for sorting out what it's really telling you, what you should do about it, and whether or not you actually care.

(There is apparently local history behind our habit of doing this, as there often is with these things. Many potentially odd habits of any particular system administration environment are often born from past issues, much as is true for monitoring system alerts. In both cases, sometimes this is bolting the stable door after the horse is long gone and not returning and sometimes it's quite useful.)

In general, this is another way of adding more context to things you may only see once in a while, much like making infrequent error messages verbose and clear. We're probably going to have forgotten about the existence of many of our automated scripts by the time they send us email, so they need to remind us about a lot more context than I might think when writing them.

PS: If you're putting a footer on the automatic email, you don't necessarily have to identify the machine by the GECOS name of its root account, but every little bit helps. And identifying the machine in the GECOS name still helps for things that don't naturally generate a footer, like email from cron about jobs that failed or produced unexpected output.

The problems with piping curl to a shell are system management ones

I was recently reading Martin Tournoij's Curl to shell isn't so bad (via), which argues that the commonly suggested approach of using 'curl example.com/install.sh | sh' is not the security hazard that it's often made out to be. Although it may surprise people to hear this, I actually agree with the article's core argument. If you're going to download and use source code (with its autoconfigure script and 'make install' and so on) or even pre-build binaries, you're already extending quite a lot of trust to the software's authors. However, I still don't think you should install things with curl to shell. There are two reasons not to, one a general system management one and one a pragmatic one about what people do in these scripts.

The general system management one is that to manage and maintain your system over time, you need to control what changes are made to it and insure that everything is handled consistently. You don't want someone's install script making arbitrary and unknown changes to your system, and it gets worse when that install script can change over time. The ideal thing to install is an artifact that you can save locally and that makes limited and inspectable changes to your system (if any). Good install options are, for example, a self-contained tarball that you can extract into a directory hierarchy of your choice (and that doesn't even have to be owned by or extracted by root), or a package for the standard package manager on your system that doesn't contain peculiar custom scripts with undesired side effects. An un-versioned shell script fetched from a remote end that you don't save or inspect and that will make who knows what changes on your system is a terrible idea for being able to manage, maintain, and understand the resulting system state.

The pragmatic reason is that for some reason, the people writing these install shell scripts feel free to have them make all sorts of nominally convenient changes to your system on behalf of their software. These shell scripts could be carefully contained, minimal, and unchanging (for a particular release), doing very little more than what would happen if you installed a good package through your package manager, but very often they aren't and you'll wind up with all sorts of random changes all over your system. This is bad for the obvious reason, and it's also bad because there's no guarantee that your system is set up in the way that the install script expects it to be. Of course generally 'make install' has the same problem, which is why experienced sysadmins also mostly avoid running that as root.

(More generally, you really want to manage the system through only one thing, often the system's package manager. This is the problem with CPAN and other independent package systems (althogh there are good reasons why people keep creating them). Piping curl to a shell and 'make install' are just magnified versions of it. See also why package systems are important.)