Wandering Thoughts archives


You should delete the 'User-Agent' header from outgoing email

We all know about the HTTP User-Agent header, which browsers and other web things send to web servers. The nominal purpose of this is covered in RFC 9110 section 10.1.5, and it's not terrible, but in practice websites have abused the header for years (if not decades) and the whole thing is a major mess (eg). A very long time ago, some mail clients decided that they'd advertise by adding an 'X-Mailer' header to email they sent, with their name in it. Somewhat more recently, various mail clients decided that they would do this using a 'User-Agent' header (sometimes in addition to an X-Mailer header); one common example is Thunderbird.

I have come to think that this is a bad idea and that you should configure your mail submission server to strip User-Agent (and probably also X-Mailer). First off, leaving this header in leaks information about your users to various people. With the way that the Internet has evolved, hiding this information is now the right answer, much like hiding user IPs turned out to be the right call. If you need to know client and device usage information for your own purposes, log the header value before you delete it (but understand that not all clients may add it in the first place).

(This information leaks not just to the people who your users send email to, but also to the people who operate the receiving email servers. These days that often means Google and Microsoft.)

Second, with the way that the spam filtering landscape has evolved into an unpredictable mess based in large part on opaque signals, other people's mail servers may well decide that they don't like certain User-Agent values. If your people are using one of those mail clients (possibly authentically, unlike spam that forges such a User-Agent), their email will be less likely to get through. Since not everything provides a User-Agent field in the first place, I believe that stripping it out entirely is not likely to be harmful, especially by comparison.

(You might feel that using User-Agent in this way is morally wrong, but other mail servers don't care about your feelings and anyway they may not be explicitly looking at 'User-Agent' as such. They may well be just feeding everything in as barely classified text and letting some pile of math look for correlations, so any header and any header value or part of its value that has correlations will be used.)

In my view, giving other people's large and opaque mail systems fewer reasons to consider real email from your people to be spam is a good reason all by itself. The privacy benefits just tilt the situation even more toward removing any User-Agent header that mail clients may have added.

(As a corollary, it's long since past time that mail clients stopped adding this header. No one is paying attention to it and it's a little leak of private information.)

spam/RemoveUserAgentHeader written at 22:41:44; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.