Wandering Thoughts archives

Another reason to reject spam at SMTP time: as a signal

I'm sure that people know all of the usual reasons to reject spam at SMTP time; in part they boil down to the fact that if you accept spam, the best you can do later is discard it, while rejecting at SMTP time means that if senders will find out if their email got mis-characterized as spam. You'll probably also lessen the load on your systems and have some other benefits (and avoid things like the forwarding problem). It's recently occurred to me that there's another advantage to SMTP-time spam rejection: it's at least potentially a feedback signal to email providers about the quality of what they're sending out.

In an ideal world free webmail providers, free mailing list providers, and so on would take great steps to make sure that they emit no spam (and certainly no spam that any commonly used spam scoring system can recognize). This is not an ideal world. This is not even anywhere close to an ideal world. Places like GMail, Yahoo Groups, and so on regularly host spammers and send us spam.

If we accept that spam and then quietly discard it, there is no feedback to these places that their user is generating bad email. However, if we reject it at SMTP time there is at least a chance that automated systems on the sender's end will notice that this user or mailing list is sending email that generates a lot of delivery failures and either react immediately or flag it for human attention. If nothing else, many mailing list setups will remove 'subscribers' who have repeated delivery failures.

Obviously, this only works for sending systems that have at least some interest in stopping spam emitting from them. But I'm optimistic enough to think that this actually does describe a lot of the free email places and mailing list providers. (And I've read that at least some of them do track SMTP rejections and other delivery failures and raise alarms if the levels get 'too high', whatever that is.)

Our solution to the spam forwarding problem

The spam forwarding problem is that when users forward their email to somewhere else, they forward their spam too. Since we have a spam tagging system, we'd like our users to not forward spam-tagged email; at the same time, we can't mandate this as an absolute requirement (and thus implement it centrally in the mailer).

You can certainly do this with procmail; selective forwarding is one of the things that procmail is good at. But there's two problems, one smallish and one big. The smallish problem is that this is not really forwarding, it's remailing. The big one is that using procmail for this is awfully complicated.

As it happens, I'm a big fan of making the right way also be the easy way. The easy way to do forwarding is to put an address in .forward; if we wanted it to be just as easy to forward only non-tagged email, we needed a .forward that only forwarded non-tagged email. So that's what we created in our mailer setup.

.forward-nonspam is just like .forward, but it only sees messages that aren't tagged as spam (or viruses) by our spam tagging system. If you also have a .forward, the .forward sees only spam-tagged email. If you only have a .forward-nonspam, spam-tagged email is quietly discarded into /dev/null because this is the common case (and it's certainly what we want to happen by default).

While we were nervous about adding non-standard .forward processing, I think that implementing .forward-nonspam has been a clear win in a whole bunch of ways. It's given us a very simple thing to suggest to users when we ask them to consider forwarding only untagged email (and many of them take us up on it, perhaps because it's so easy to do). It's even simplified some of our own .forward use.

(For technical reasons that don't fit within the margins of this entry, it's not quite easily useful for the case of 'put all non-spam stuff in my regular inbox and all spam stuff into a file that I can look at every so often if I want to'.)

Another reason to allow mail origin address forgery

I've written before about my desire that Unix mailers would allow more forgery of outgoing email addresses, back in PleaseAllowForgery. At the time I was thinking of direct usage for sending email yourself, but I've recently realized that there's another important use for this: remailing things from forwarding setups.

If you just want simple forwarding, you can do that in .forward. But if you want more sophisticated conditional forwarding, around here the tradition approach is to use procmail with a suitably complex .procmailrc that winds up forwarding selected email with procmail's ! directive. So far, so good.

The problem is that this doesn't forward the message as such; instead it causes procmail to re-mail it. When it's remailed, it gets your own address as the envelope origin address. Should your remailed message bounce, which has been known to happen, the bounce will go to your address where it will be re-processed by your procmail setup and quite possibly 'forwarded' to your other address again. Repeat until the mail system melts down.

(These days our mail system has a circuit breaker that tries to detect and cut off such bounce loops, but it's not a sure thing.)

If Unix MTAs were more willing to allow you to set the envelope origin address, it would be entirely sensible for procmail and similar programs to do so when they 'forwarded' email. This would avoid the looping bounce problem (now bounces would go to the original sender, although this has its own problems) and make procmail-based forwarding much more like just using a simple .forward setup. As a bonus, it would avoid the general problems with remailing since, well, things wouldn't be remailing any more.

(Possibly some procmail alternatives already try to do forwarding this way, and leave it up to the MTA to decide whether it works or not. If so, I applaud them.)

Sidebar: the forwarding bounce problem

The problem with bounces of forwarded email is that there is no good place to report the problem. Even if the sender address isn't forged, the sender can't do anything about broken forwarding, so the most that sending bounces to them does is let them know that their message didn't get through. Sending the bounce to the user's local mailbox is likely pointless because the user probably doesn't check their local mailbox since, after all, their email is being forwarded. Trying to send any sort of note to the forwarding address runs into the issue that, well, at least some email to it is already bouncing.