Spam issues need to be considered from the start
A number of Google's issues from the spammer I talked about yesterday come down to issues of product design, where Google's design decisions opened them up to being used by a spammer. I considered these issues mistakes, because they fundamentally enable spammers, but I suspect that Google would say that they are not, and any spam problems they cause should get cleaned up by Google's overall anti-spam and systems that watch for warning signs and take action. Well, we've already seen how that one works out, but there's a larger problem here; this is simply the wrong approach.
In a strong way, anti-spam considerations in product design are like (computer) security. We know that you don't get genuinely secure products by just building away as normal and then bringing in a security team to spray some magic security stuff over the product when it's almost done; this spray-coated security approach has been tried repeatedly and it fails pretty much every time. The way you get genuinely secure products is considering security from the very start of the product, when it is still being designed, and then continuing to pay attention to security (among other things) all through building the product, at every step along the way. See, for example, Microsoft's Security Development Lifecycle, which is typical of the modern approach to building secure software.
(That you need to take a holistic approach to security is not really surprising; you also need to take a holistic approach to things like performance. If no one cares about performance until the very end, you can easily wind up digging yourself into a deep performance hole that is very painful and time-consuming to get out of, if it's even feasible to do so.)
Similarly, you don't get products that can stand up to spammers by
designing and building your products without thinking about spam,
and then coming along at the end to spray-coat some scanning and
monitoring magic on top and add an
abuse@... address (or web
form). If you want products that will not attract spammers like
ants to honey, you need to be worrying about how your products could
be abused right from the start of their design. By now the basics
of this are not particularly difficult, because we have lots of
painful experience with spammers (eg).
Google is objectively running a spammer mailing list service
If you are a mailing list service provider, there are a number of things that you need to do, things that fall under not so much best practices as self defense. My little list is:
- You shouldn't allow random people you don't know and haven't carefully authenticated to set up mailing lists that you'll send out for them.
- If you do let such people set up mailing lists, you should require that all email addresses added to them be explicitly confirmed with the usual two step 'do you really want to subscribe to this' process.
- If you actually allow random people you don't know to add random email addresses to their mailing lists, you absolutely must keep a very close eye on the volume of rejections to such mailing lists. A significant rate of rejections is an extremely dangerous warning sign.
Google, of course, does none of these, perhaps because doing any of these would require more people or reduce 'user engagement', also known as the number of theoretical eyeballs that ads can be shown to. The result is predictable:
2017-10-04 08:19 H=mail-io0-f199.google.com [18.104.22.168] [...] F=<emails1+[...]@offpay.party> rejected [...] 2017-10-04 08:26 H=mail-ua0-f200.google.com [22.214.171.124] [...] F=<emails5+[...]@offpay.party> rejected [...] 2017-10-04 08:31 H=mail-vk0-f71.google.com [126.96.36.199] [...] F=<emails7+[...]@offpay.party> rejected [...] 2017-10-04 08:31 H=mail-pf0-f198.google.com [188.8.131.52] [...] F=<emails7+[...]@offpay.party> rejected [...] 2017-10-04 08:32 H=mail-qk0-f198.google.com [184.108.40.206] [...] F=<emails8+[...]@offpay.party> rejected [...] 2017-10-04 08:39 H=mail-qk0-f199.google.com [220.127.116.11] [...] F=<emails9+[...]@offpay.party> rejected [...] 2017-10-04 08:40 H=mail-it0-f70.google.com [18.104.22.168] [...] F=<emails9+[...]@offpay.party> rejected [...] 2017-10-04 08:40 H=mail-io0-f200.google.com [22.214.171.124] [...] F=<emails11+[...]@offpay.party> rejected [...] 2017-10-04 08:40 H=mail-io0-f197.google.com [126.96.36.199] [...] F=<emails11+[...]@offpay.party> rejected [...] 2017-10-04 08:41 H=mail-ua0-f197.google.com [188.8.131.52] [...] F=<emails11+[...]@offpay.party> rejected [...] 2017-10-04 11:57 H=mail-vk0-f69.google.com [184.108.40.206] [...] F=<emails15+[...]@offpay.party> rejected [...] 2017-10-04 12:06 H=mail-pg0-f71.google.com [220.127.116.11] [...] F=<emails16+[...]@offpay.party> rejected [...] 2017-10-04 12:09 H=mail-qt0-f200.google.com [18.104.22.168] [...] F=<emails18+[...]@offpay.party> rejected [...]
That's just from today; we have more from yesterday, October 2nd,
and October 1st. They're a mixture of
RCPT TO rejections (generally
due to 'no such user') and post-
DATA rejections from our
commercial anti-spam system laughing very
loudly at the idea of accepting the email. Many other copies made
it through, not because they weren't seen as spam but because they
were sent to users who hadn't opted into our SMTP time spam rejection.
Google has deliberately chosen to mix all of its outgoing email into one big collection of mail servers that third parties like us can't easily tell apart. For Google, this has the useful effect of forcing recipients to choke down much of Google's spam because of GMail, instead of letting people block it selectively. In this case, we have some trapped mail headers that suggest that this is something to do with Google Groups, which is of course something that we've seen before, with bonus failures. That was about six years ago and apparently Google still doesn't care.
(Individual people at Google may care, and they may be very passionate about caring, but clearly the corporate entity that is Google doesn't care. If it did care, this would not happen. At a minimum, there would be absolutely no way to add email addresses to any form of mailing list without positive confirmation from said addresses. Instead, well, it's been six years and this stuff is still happening.)
PS: My unhappy reactions yesterday on Twitter may have produced some results, which is better than nothing, but it should go without saying that that's not exactly a good solution to the general issue. Spammers are like ants; getting rid of one is simply dealing with the symptoms, not the problems.
If spam false positives are inevitable, should we handle them better?
Right now, our mail system basically punts on handling false positives (non-spam detected as spam). The only thing that users can do about false positives is turn off SMTP time rejection (if they have it turned on) and then fish the mis-classified message out of their filters or our archive of all recent email they've gotten. If the message has already been rejected, the only thing that can be done is to get the sender to re-send it. And there's no way for users to see what messages have been rejected, so they can tell if some important email message has fallen victim to a false positive; instead we periodically get requests to check our logs.
My impression is that our mail system's behavior is not atypically bad, and instead that plenty of other mail systems behave in much the same way. It's pretty straightforward to see why, too; it would take significantly more work to engineer anything more than this, especially if you reject at SMTP time (and I think you want to, because that way at least people find out that their email hasn't gone through because of a false positive). But probably we should do better here, if only because this is a pain point for our users (it's one of the things that gets them to talk to us about our spam filtering).
(This is also probably required if we accept the idea that 'spam levels' may be a copout.)
In general, a mail system do things with potential false positives
from two sides. It can give the local receiving user some tools to
investigate situations, answering the question of 'did X try to
send me email and what happened to it', and perhaps also to retrieve
such mis-classified email. Retrieving email rejected at SMTP time
requires your mailer to save a copy of such email (at least for a
while), which means you need to defer rejections to
This opens up a complicated tangle of worms for messages sent to
multiple recipients (although they go away again if you mandate
that you only have one 'spam level' and
everyone gets it).
Your mailer can also give senders some tools they can use to cause false positive messages to get accepted anyway. You probably don't want to offer these tools to all senders; sure, most spammers aren't paying attention to you, but some spam (such as advance fee fraud attempts) does come from real human beings doing things to compromised mail systems by hand, and they might well take advantage of your generosity. However, if someone's regular correspondent has some email classified as spam, it's probably safe (and worthwhile) to offer them these tools. The odds are probably good that it's an accident as opposed to a compromised account with a human being at the other end to take advantage of you.
(There are a wide variety of options for how to let people psuh
messages through. You could do 'mail it to this special one-time
address', or 'include this magic header or
Subject marker', or
just 'visit this URL of ours to request that your email get delivered
anyway'. And I'm sure there's more. I have no idea which option
would work best, and SMTP-time rejection makes things complicated
because it's hard to give people much information.)
None of these are particularly easy to put together with off the shelf components, though, which for many places is probably going to make the entire issue moot. And maybe things should be left as they are for the straightforward reason that a low level of false positives just doesn't justify much sysadmin effort to improve the situation, especially if it requires a complicated bespoke custom solution.
(This is one of the entries where it turns out that I don't have any firm conclusions, just some rambling thoughts I want to write down.)
The idea of 'spam levels' may be a copout
I recently wrote about using the spam scores from another mail system at the university. In a comment, Robert Sander suggested that the original email system should have just rejected the spam at SMTP time. There are a number of issues here, but one of the traditional reasons not to do this is to provide your users with varying levels of spam filtering (which is something we do). This is a perfectly traditional reason, but perhaps this is a copout answer (to be fair, an unexamined one).
The fundamental problem with the idea of spam levels is the usual problem with asking people questions, namely that most people aren't going to be able to make useful decisions because they don't have enough knowledge. With spam scoring levels this is even more of an issue than usual, because many of the answers are either unknowable or take statistical analysis to answer properly. For example, the only reason that I know something about the distribution of the spam scores in our incoming email is because I've gone well out of my way to do analysis on our logs (and I have access to those logs). If I were to ask a user to choose between rejecting on a '70%' spam rating and an '85%' spam rating, how on earth are they supposed to make a sensible choice? At a minimum they'd need to figure out the distribution of spam scores for their email, both legitimate and spam, to see if this is a useful or sensible choice.
In practice there's only one thing that users are going to do with a spam levels knob. They're going to make it more aggressive when they get annoyed with spam and then if they find out that they had false positives (rejecting real email that they want), they might reluctantly make it less aggressive again. Even this represents a failure on our part, since an ideal mail system shouldn't require this tuning in the first place for almost all users.
(The exception are users who get messages that look a lot like spam but aren't. These users will probably always need some way to let through more questionable things.)
So I think there's a serious argument that features like 'spam levels' are essentially a copout. They're our attempt to wash our hands of taking responsibility for the unsolvable problem of rejecting all spam without rejecting anything else. Sure, we can't solve the problem, but we owe it to our users to give it our best shot and then own up to the resulting imperfections as the best tradeoffs we can achieve. Making and justifying this sort of tradeoff is part of what system administration is about in general, after all.
(If we do this, we might also want to think seriously about how we can deal with false positives in an environment where the answer is not 'well, turn your spam filtering off'. Sadly, this probably involves more sophisticated filtering and scoring than is provided by the commercial anti-spam package we use.)
(Some years ago I wrote that filtering spam for users was part of our job, which sort of touches on the same ideas.)
People probably aren't going to tell you when your anti-spam systems are working
As part of yesterday's entry about how we're now using the spam scores generated by the university's central email system, I was going to say that our users seem happy with the results. However, I have to admit that this is not quite true. We don't explicitly know that they're happy; instead, what we know is that they've stopped reporting that they're getting too much spam and can we do something about that.
What I've come to expect is pretty straightforward, namely that users generally aren't going to give us feedback when our anti-spam systems are working well. And why should they? Really, a spam-free email system where all the email they want gets delivered with no false positives is just how things should be, and you don't generally tell people 'good job, the systems are working just how they're supposed to work'. Naturally, people are generally only going to tell you when something goes wrong, either what they think of as an excess of spam or when email they're expecting doesn't get through or gets bounced.
On the one hand, this can be a bit frustrating when (or if) we want to know if some theoretically clever trick we've added to the mail system is making people's email more pleasant. On the other hand, this means that no news is good news; if we're not getting complaints about spam or missing email, we're most likely (still) doing things right. If we change something and nobody says anything, at a minimum our change did no harm.
As a side note, it's probably not reliable to count on users to start complaining if (or when) the amount of spam they see goes up. By now, some number of users have been trained to expect a certain amount of spam in their inboxes, and they won't start complaining out loud until the spam getting through really gets excessive.
(They may click on 'this is spam' or 'mark as spam' buttons if those buttons are conveniently available to them in their mail environment, especially if the buttons appear to do something. If you have such buttons, monitoring how often users click them can likely give you an early warning indicator of increased spam getting through your filters. Or at least email that your users don't want.)
We've wound up using the spam scores from some other mail systems
Like many places, the university has a central email system and all staff and professors have an address there. One of the things you can do with your UTORMail email is forward all of it to an email account elsewhere, such as your account here. A decent number of our users both do this and make a reasonably significant use of their central email address, making it visible and active and thus periodically hit by spammers. When these central email addresses get hit with spam, the central email system forwards it on to us.
Starting a couple of months ago, the university's central email seems to have been targeted by a number of active spam campaigns. People who read their email in the central system were commenting on it and certainly some of our users were reporting it to us, because there was a problem; the commercial anti-spam package we use wasn't recognizing this spam as spam. In theory we could give our normal answer of 'it's a black box, we get what we get', but in practice this felt unsatisfying because the central email system was recognizing and tagging this email as spam before it passed it on to us.
Since the central email system actually uses the same commercial
anti-spam package that we do, my assumption was that our lower spam
score was because we weren't directly receiving the spam. Since it
had gone through the central email system, there was a layer or two
Received: headers and other obscuring things (and the source
IPs were different for DNS blocklist checks and so on). It was
fairly obvious spam so we were scoring it relatively high, but after
the layer of forwarding it wasn't quite sufficiently obvious to get
scored as definitely spam.
This was kind of frustrating. The central email system was putting its
spam scoring information right there in the message headers of the
forwarded messages; we just weren't paying any attention to it. Well,
we could change that, so we did (which took a little bit of work in
Exim). Now, if we don't score something as spam but the central email
system does, we still mark it as spam in the
Subject: header, which
triggers various downstream processing. In practice, users who are doing
spam filtering at all will have this forwarded email go away just the
way regular spam does.
(We considered doing something more sophisticated and selective in
Exim, but decided that there were simply too many places in our
overall mail setup that knew about the
Subject: tag, including
filtering done by users through
procmail and so on. Also, we
couldn't think of anything we'd want to do differently depending
on who had determined it was spam.)
Since we don't allow a 'not-spam' score from the central email system to override our own opinion, we don't make any attempt to limit this special handling to email that we definitely received from the central email system. This has the useful side effect that if you're forwarding your email through another system before it gets to us, we'll still pay attention to the central email system's spam scoring for you.
The types of attachments we see for malware-laden email
These days we block some attachment types, but of course not all of them. We also use our commercial anti-spam package to detect and reject malware that isn't already rejected outright as a bad attachment type. This raises the obvious interesting question of what attachment types that malware comes in. Unfortunately I can't answer exactly that question from our logs, but I can see what attachment types we see in email that is rejected because it contains malware.
(This difference matters only if a malware email contains multiple attachments, possibly with multiple pieces of malware. When this happens, our commercial anti-spam system doesn't log information that would let us identify which particular attachment has the detected malware.)
In the past 19 days of logs, we rejected 440 messages because they were identified as containing malware. A few of these messages had attachments without MIME filenames; all of the attachments were detected as either Microsoft Word files or images (mostly PNGs with a couple of GIFs). The vast majority had MIME filenames, and the (claimed) extensions on these are as follows:
241 .doc 95 .jar 52 .zip 24 .html 6 .htm 6 .gz 3 .7z .rar 1 .z .xls .rev .psx .pdf
Most of the .zip files contained .doc files (26) or .jar files (12). Ten of the .zip files were actually given the MIME type application/msword (despite the .zip extension in their MIME filename) and contain the file extensions '.bin .png .rels .xml none'. Some Internet searches suggest that the .bin file extension here is the giveaway marker of Word macros being embedded in the file, and we certainly saw a bunch of .doc files that had .bin files inside them (it looks like 69 out of the 241 .doc files, so by no means all or even a majority but certainly a significant portion).
Having gone through this stats gathering exercise, my overall view is that there's nothing terribly surprising here. Microsoft Word and Java/JARs are a big attack vector and HTML files appear to be a side door to sneak some attacks and phishing through where they might be blocked as an actual HTML email message.
(The obvious question to ask next is if our users get very many legitimate email messages that have Word files with embedded macros.)
Some people feel that all permanent SMTP failures are actually temporary
It all started with a routine delivery attempt to my sinkhole SMTP server that I use as a spamtrap:
remote 22.214.171.124:36462 at 2017-07-03 13:30:28 220 This server does not deliver email. EHLO mail.travelshopnews7.com [...] MAIL FROM:<firstname.lastname@example.org> 250 Okay, I'll believe you for now RCPT TO:<redacted@redacted> 250 Okay, I'll believe you for now DATA 354 Send away [...] . <end of data> 554 Rejected with ID 433b5458d9d3e8a93020aca44406d2ec1d8ba82a QUIT 221 Goodbye
That ID is the hash of the whole message and its important envelope
information (including the sending IP). So far, so normal, and these
people stood out in a good way by actually
QUITing instead of just
dropping the connection. But then:
remote 126.96.36.199:39084 at 2017-07-03 13:37:10 220 This server does not deliver email. EHLO mail.travelshopnews7.com [...] 554 Rejected with ID 433b5458d9d3e8a93020aca44406d2ec1d8ba82a
They re-delivered the exact same message again. And again. And
again. In less than 24 hours (up to July 4th at 9:28 am) they did
21 deliveries, despite getting a permanent refusal after each
At that point I got tired of logging repeated deliveries for the
same message and put them in a category of earlier blocks:
remote 188.8.131.52:50676 at 2017-07-04 10:38:10 220 This server does not deliver email. EHLO mail.travelshopnews7.com [...] MAIL FROM:<email@example.com> 550 Bad address RCPT TO:<redacted@redacted> 503 Out of sequence command [...]
You can guess what happened next:
remote 184.108.40.206:34094 at 2017-07-04 11:48:36 220 This server does not deliver email. EHLO mail.travelshopnews7.com [...] MAIL FROM:<firstname.lastname@example.org> 550 Bad address RCPT TO:<redacted@redacted> 503 Out of sequence command [...]
They didn't stop there, of course.
remote 220.127.116.11:53824 at 2017-07-13 15:37:31 220 This server does not deliver email. EHLO mail.travelshopnews7.com [...] MAIL FROM:<email@example.com> 550 Bad address RCPT TO:<redacted@redacted> 503 Out of sequence command [...]
Out of curiosity I switched things over so that I'd capture their message again and it turns out that they're still sending, although they've now switched over to trying to deliver a different message. Apparently they do have some sort of delivery expiry, presumably based purely on the message's age and totally ignoring SMTP status codes.
(As before they're still re-delivering their new message despite the
DATA permanent rejection; so far, it's been two more deliveries
of the exact same message.)
These people are not completely ignoring SMTP status codes, because they know that they didn't deliver the message so they'll try again. Well, I suppose they could be slamming everyone with dozens or hundreds of copies of every message even when the first copy was successfully delivered, but I don't believe they'd be that bad. This may be an optimistic assumption.
(Based on what shows up on www.<domain>, they appear to be running something called 'nuevoMailer v.6.5'. The program's website claims that it's 'a self-hosted email marketing software for managing mailing lists, sending email campaigns and following up with autoresponders and triggers'. I expect that their view of 'managing mailing lists' does not include 'respecting SMTP permanent failures' and is more about, say, conveniently importing massive lists of email addresses through a nice web GUI.)
LinkedIn is still trying to send me email despite years of rejections
Back in 2014 I wrote about LinkedIn sending me invitation spam emails and how they wanted me to agree to their terms of service (and join LinkedIn) in order to 'unsubscribe' from them. Of course I didn't do that; instead, as usual, I arranged to have all future email from LinkedIn to me to be rejected during the SMTP conversation on our external MX gateway (using one of our anti-spam features). Then I put the whole thing out of my mind.
You can probably guess what has happened since then. It's now
closing in on three years that I've been rejecting all such LinkedIn
email, and LinkedIn still attempts to send me some every so often
on a semi-regular basis. I have no idea what's actually in the
email, since the external MX gateway rejects it at
RCPT TO time
(and LinkedIn uses completely anonymous
MAIL FROM addresses), but
I suspect that it's more invitations.
Persistently sending email to addresses that fail at
RCPT TO time
makes LinkedIn's behavior functionally indistinguishable from
spammers. Spammers ignore
RCPT TO and other mail failures; so
does LinkedIn. Spammers will send to dead addresses for years.
LinkedIn? Check. I am sure that LinkedIn will claim that it has
good reasons for its behavior, and perhaps it will even allege that
it is merely doing the will of its users. It doesn't really matter.
When you walk like a duck and quack like a duck, people who don't
want ducks don't really care what you actually are (cf).
I believe that LinkedIn's behavior is illegal in Canada under our anti-spam legislation. I was going to say that this exposes LinkedIn to potential legal risks now that it's 2017 and the legislation is fully in force, but it turns out that the government suspended the right of private action recently. Since Canada is a loser-pays country for civil lawsuits, suing LinkedIn over this would always be risky, but now only the government can take them to court and I don't think that that's very likely.
(On the other hand, according to the website the government apparently has taken action against some big Canadian corporations over their spam, oops, 'marketing email'. So who knows.)
PS: These days there appears to be a LinkedIn unsubscribe page that doesn't immediately demand that you log in to LinkedIn. I haven't tried it; to put it one way, I don't particularly believe that leopards actually change their spots. I have no trust for LinkedIn at this point and thus no desire to actively provide them with any email addresses.
The TLDs of sender addresses for a week of our spam (June 2017 edition)
Once upon a time the Internet only had a few non-country top level domain names. Then that changed. Mostly these new TLDs get used for websites, but every so of people use them for email. Generally the stereotype is that it's mostly spammers using these new TLDs, so I thought it would be interesting to look at eight days worth of logs from our commercial anti-spam system to see what the TLDs of sender addresses looked like for messages that were scored as spam and messages that weren't.
So here are the top ten TLDs from email scored as spam, with the percentage of our spam-scored email that had a sender address in that TLD and what percent of the TLD's overall email the spam represents.
|TLD||% of total spam||spam as % of TLD|
We can immediately see that
.bid does terribly and
.us is not
doing so well. The
.bid spam comes from multiple domains and
probably multiple spammers (there are at least two or three patterns
in how the sender addresses are formed).
.info is close to as
.us, but it's a much smaller percentage of the email.
.us spam seems to be a mix of compromised
random domains, and active spammer domains. The
.info spam is
multiple domains but might be mostly one spammer.
The high popularity of
.com in spam sender addresses surprises
me, as does how much of
.com email is spam. Bear in mind that
we're a university department (and in Canada), so we probably
exchange much less normal email with
.com places than most
However, the new TLDs are not particularly popular with spammers. Even if I look all the way down in the data, it's dominated by country codes with only a few new TLDs in small quantity:
|new TLD||% of spam|
You get the idea. I haven't shown 'spam as a percentage of the TLD's
email' here because it's mostly 100% and the times when it's not,
it may be because of mis-scoring (the absolute numbers are very
small, so it doesn't need much mis-scoring to show up as an appreciable
.party is under a hundred messages over the eight days of
.biz sender addresses are only 79% spam as
scored by our system.
Pleasingly, there were exactly 200 different TLDs used in the logs (or 199 if you exclude the null sender, which was 0.3% of the spam and 56% spam).