Wandering Thoughts archives

Why Google's handling of multiple domains on inbound messages is okay

It started with a tweet by @xlerb (Jeb Davis):

Today I learned that Google thinks they can unilaterally redefine SMTP: <link> (warning: gratuitously shouty forum comments)

To summarize the link: Google is the MX target for all sorts of domains, due to various services they offer. Google's MX servers will now only accept destination addresses in a single domain per transaction; if you try to RCPT TO to addresses at multiple (Google-hosted) domains in the same transaction, all but the first domain will get 4xx temporary failures.

I'm not particularly fond of Google's handling of email, but as I tweeted I come down on Google's side here. First off, this can't possibly be called 'redefining SMTP'. Mail servers have always been allowed to temporarily defer some recipients for any reason whatsoever, including random software limitations, their own convenience, and obscure internal policies. Anyone who expects all recipients to always be accepted on the first delivery attempt has not been paying attention to the modern Internet mail environment for years; many, many systems behave otherwise (ours included). The only vaguely novel thing Google is doing is that they are being clear about why addresses are getting temporary failures.

(It would be redefining SMTP if Google was giving 5xx permanent failures in this case and telling everyone to fix their software to not do this, but not even Google is that stupid.)

Second, there is an excellent reason why Google might want to do this; it is my old friend the lack of partial success for message delivery. If different Google-hosted domains can have different policies on what message contents can be sent to them (perhaps Google allows the domain owners to control this), Google needs to make sure that it's never in a situation where some recipient domains would accept the message but others would refuse it. Giving each domain its own MX IP address (or set of them) is not exactly a scalable solution (not at Google's scale), so Google has to do it the other way; they can only ever accept a single domain per transaction, so only a single domain's policy will apply to the message contents.

Finally, my view is that any significant mailing list operation that's having problems about this is probably doing things wrong. For a start, any mailing list using VERP will not be affected by this, because with VERP each transaction has only a single recipient. And you should really, really be using VERP and automated bounce handling if you're running a mailing list of any appreciable size.

(Note that in theory, ordinary people can run into this routinely; all you need to be doing is having a conversation with several people who are all hosted through Google but at different domains. Depending on how fast your mail system does retries, some of the people in the conversation may get messages much slower than others. In practice, who knows what special magic Google is doing.)

All of this is something that goes well beyond what Google is doing right now. Every mail server that wants to make accept/reject decisions based on the both the message contents and the destination addresses (or domains) faces this issue, and there are no better solutions than what Google is doing. If you want to allow people or hosted domains to reject during SMTP (and you do), and you want to give them some control over what gets rejected, you're going to wind up doing the same thing.

(And you should not feel particularly broken up about it. Batching multiple addresses with different destination domains together into a single transaction when they all MX to the same thing is an optimization, not a fundamental feature of SMTP. It just happens to be a common optimization in mailers, partly because it's cool in a way that attracts programmers like bees to honey.)

Some amusing cut and paste work from spammers

Recently I got a modest spate of advance fee fraud spam attempts with the interesting feature that they either claimed to be from 'Federal Bureau Of Investigation Seeking To Wiretap The Internet' or at least contained some variant of 'FBI seeking to wiretap the Internet' in addition to the agency name. Advance fee fraud come-on messages are almost never well written to start with but this text is relatively glaringly out of place, which is part of why it stood out and stuck in my mind. The messages have some other similarities but also their fair share of differences, so I'm not sure I can conclude that it's the work of a single group (I suspect that advance fee fraud spammers aggressively copy from each other's come-on messages). My records say that this is not a new thing; the oldest sample I could spot using a quick search pattern dates from the end of 2008.

(It looks like an Internet search for this phrase will turn up lots and lots of archived samples.)

What interests me is speculating on where this odd text comes from and what it implies about how spammers operate. In general, the 'seeking to wiretap' text is clearly out of place in the spam messages; there is no attempt to weave it into the come-on text and it's generally more or less positioned as part of the FBI's name. The obvious guess about what happened is that at some point an initial spammer was looking for the FBI's full name, did an Internet search, and wound up on a news story where this text was the main heading or the like instead of the FBI's own page. Operating without enough contextual knowledge they lifted the entire text, copied it into their spam, and it propagated from there. That the text continues to show up with some regularity suggests that it's become established in some mainline of advance fee fraud messages that lots of people copy from.

This is where I start thinking of similarities to evolutionary biology, where odd and unimportant features of a successful organism can sort of come along for the ride as it propagates. This bit of text feels like one of them; I doubt that it itself does anything to improve the spam's success rate, but it could well be part of a relatively successful initial advance fee fraud message that has been widely copied and imitated more or less wholesale since then. This is especially so because the text usually appears as an initial title block and I can certainly believe that those just get copied back and forth without anyone paying them much attention.

(While there are theories that advance fee fraud spammers deliberately make their come-on messages relatively extreme and obvious in order to hook only the most credulous, I don't believe that this text is being included deliberately as part of that filtering. To use the text as filtering seems more than a little bit too subtle and clever for both the spammers and the audience they are allegedly filtering for.)