Wandering Thoughts archives

The evolution of filtering (a story from the Usenet era)

When I first started reading Usenet (back in the rather old days), I wound up hanging around some old Unix and Usenet hands. One of the things that I noticed about how they read Usenet was that they had really aggressive killfiles; they would go into what I considered a high quality technical newsgroup like comp.unix.wizards and their newsreader would promptly junk all but one or two of the articles.

At the time, I did not understand this at all. Sure, there was a certain amount of pointless chaff even back then, but they had to also be throwing out huge amounts of solid, interesting stuff; how could they stand to miss them?

I took the opposite attitude. For a long time I didn't use a killfile at all, because I didn't want to run the risk of accidentally missing a good article. Even when I started using killfiles, I was very conservative about what I junked; I felt it was better to wade through some amount of chaff rather than miss a gem.

Then one day I woke up and realized that I was wrong, because I had been fooling myself about the actual choices. In reality it was not a choice between reading only some of the gems or reading all of the gems (and some amount of chaff); it was a choice between effectively reading no gems at all and reading at least some, even if I missed others. Once I realized this, my Usenet filtering gradually became more and more aggressive. As the noise increased in the newsgroups I read, I became more and more willing to filter out almost everything in order to at least get something or at least not waste lots of time for nothing.

There is an obvious extension of this to email, and in fact I've wound up feeling that the same dynamic is inevitable. Put one way, there is only so much noise that people will put with for so long. At some point I expect people to switch attitudes the way I did way back when, and stop feeling that every email is (potentially) precious in favour of an attitude of extracting what use they can from email, even if it's imperfect and incomplete.

(Existing spam filtering is already helping with this, because it means that fewer and fewer people consider email a reliable communication method any more. 'It got eaten by my spam filter' is a great excuse partly because it's so often true.)

Dear Unix mailers: please allow more forgery

Here is a peculiar irritation I have with Unix mailers (the combination of both MTAs and MUAs): I wish they made it easier for people to forge their outgoing email address, the address that appears as the From: and in the envelope sender and so on.

Unix mailers generally make it at least possible and sometimes easy to do a half-assed job of forgery. You can put in your own From: and many mailers will leave it alone, but they'll then also add a Sender: header and usually won't touch the envelope sender address. When I am in a cautious mood, this is not good enough; I want no traces in the headers that an outsider could easily use to identify my email address or even to identify that the From: address is not a fully real address.

Most IMAP-based MUAs make this thorough level of forgery so trivial that it's not even forgery, it's just multiple identities. I have four different ones configured in Thunderbird and the most difficult bit of setting them up was knowing that the 'manage identities' button was the Preferences option that I wanted (and Thunderbird is smart about using them, too). But Unix mail systems are (or at least seem to be) pointlessly stubborn about this.

(Please don't suggest that I abandon my Unix mailer for some IMAP client. I am very attached to my Unix MUA of choice and it's probably the single oldest piece of my Unix environment by now.)

In today's world of spam and untrusted destinations there are lots of reasons to want to thoroughly use other sender addresses, and given that you can already do this through other mechanisms I think that Unix mailer environments should at least default to allowing this and making it relatively easy.

(This should not affect your ability to trace email back to its actual sender if someone complains; just put the UID that the email was received from in the Received: headers or the like. I believe that this is more or less the default for Unix MTAs these days.)

PS: since I only use one Unix MUA and it's an uncommon choice, maybe all of the common ones have changed over to doing what I want and I'm really just griping about a single MUA.