Wandering Thoughts archives

The status of null-sender spam from outlook.com

Recently, David left a comment on my last entry on null sender spam from outlook.com noting that his site had seen a stop of null sender spam from Outlook at the end of December. This made me curious about what we're seeing (and David asked, too), so I've now gone looking.

The short version is that clear null sender spam from outlook.com appears to have stopped at the end of last year (and I mean literally the end of last year, as we have entries from December 31st). We're still getting some amount of email from outlook.com with null sender addresses, but our anti-spam system now scores all of it very low. I can't be sure that this isn't spam, but it's certainly entirely possible that it's real bounces. We continue to get spam from outlook.com in general; at the moment, our 2016 figure is that about 4% of email from outlook.com scores high enough to be considered spam. In December the logs say it came out to be about 11.5% spam, so we clearly saw a significant drop here.

David also reported a lack of general spam from outlook.com. Unfortunately we don't see that. Outlook.com has been consistently sending us some amount of spam (as scored by our systems). In addition, several outlook.com hosts are currently on the SBL; out of microsoft's listings, I can spot more than five listings. However the SBL seems to be doing something odd here, in that they're listing .0 addresses in the /24 instead of the actual IP address they list in the SBL listings. The net effect is that nominal SBL listings won't actually block anything, which kind of irritates me.

(Eg, SBL273948 says 'Spam source @104.47.100.234' but is for 104.47.100.0/32.)

My overall view is that outlook.com continues to have a spam problem, but they have apparently managed to block or otherwise stop one source of their spam. This is progress; it is just not enough progress. Having roughly one in twenty email messages that we receive from you being spam is not a good ratio. For scale, over the same period in 2016, only 0.2% of the email received from Google was scored as spam.

(This includes both GMail email and email from some other things at Google that send out email, since as far as I know you can't tell the email servers apart, assuming there even is different infrastructure for the various different email systems.)

SMTP submission ratelimits should have delays too

We've recently enabled ratelimits on our mail submission servers. By this I mean that when you hit the ratelimits (which are applied to recipients, ie RCPT TO commands), the mailer gives you 4xx replies and your client will probably error out. But when I set up the ratelimits, I didn't just do that; I also made the mailer delay for 10 seconds before responding to every ratelimited RCPT TO. It's my view that this is a useful and in fact important thing to do.

You should normally only have ratelimits on submission in order to deal with spammers, which means that when the ratelimits trigger you hopefully have a spammer. One of the things you want to do with spammers is slow them down; the slower the spammer is going, the less messages they're submitting and the less damage they're doing. If you immediately reject RCPT TOs, you're slowing down the spammer in one sense (their message isn't getting through to very much), but not in another sense; their submission client will get 4xxs at a rate of many a second and finish message submission almost immediately. They can try again immediately or switch over to another compromised account they have or whatever. Delaying before each reply slows the spammer's submission client down, possibly significantly. If you delay for 10 seconds (as we do), now the spammer can only test 6 RCPT TOs a minute. Unless their code abandons message submission when it goes too slowly, you're tying up their submission code for tens of minutes for a typical spam message with a mass of recipients, instead of letting them dump the message on you and be rejected in a second or so.

(The other benefit of slowing the spammer down is that it gives your alerting mechanisms more time to light up with alarms and get your attention.)

Now, it's certainly possible that this frustrates spammers less than I think. Perhaps their software abandons message submission on the first 4xx reply and immediately moves on (including abandoning the current account). But on the other hand it's a cheap precaution and I think it's worth a shot.

(A spammer that does many message submissions in parallel can speed things up here, but then they open themselves up to other limiters. It's almost certainly rather unusual to have multiple mail submission sessions running at the same time, especially once ratelimiting has kicked in.)

Your outgoing mail system should have a per-sender stop switch

Here is something important we have come around to realize as one result from recent events. Put simply, every system that handles outgoing user-generated email should have some method to immediately stall and stop all email from a specific user. You want this for the obvious reason; when you discover you have a compromised user account that's being used to send spam, you can immediately stop just their email instead of having to take down your entire outgoing email system.

When you implement this, don't just implement the obvious step of refusing email being submitted by blocked user(s). Go the extra distance so that blocking a user also (immediately) stops further processing of any email from them that you have in the mailer's queues. Generally, by the time you detect that a user's been compromised and their account is being used to spam, you're going to have a bunch of email from them queued up in your system trying to get delivered to various places. You really don't want to have to hunt all of this email down by hand to stop it from being sent out; instead, it's much more better if you put a login name or whatever in a control file and all the queued email from them stops dead, with no fuss or muss.

(I wouldn't automatically remove such email from the queue, partly because you may want to inspect sample messages. It's enough that the messages stop trying to be delivered without you having to stop the entire mail system in a panic.)

Usually you'll want to do this based on the authenticated user (generally from SMTP AUTH). In some environments people don't have to authenticate to your outgoing mail server; here the best you can do is base things on, eg, the MAIL FROM. If you're dealing with this situation, you may want to support wildcards (so you can say 'all email with a Hotmail sender address gets stopped'). Spammers often but not always use revolving MAIL FROM addresses where they think they can get away with it, so you need to be prepared for that.

(You may also want to support a per-IP or per-subnet stop switch, especially if you don't have SMTP AUTH to attach reliable identities to submitted email. But things are starting to get intricate here and at some point you're better off just stopping the entire mail system and doing general searches through the queued email.)

We need to deploy anti-spam precautions even if they're a bit imperfect

A few years ago we had a local spam incident. In its wake, we made some configuration changes and started exploring things like ratelimiting outgoing email. Our first step in this was to set our Exim configuration to track rate limits without enforcing them, so that we could figure out what limits to set that would stop spammers without causing problems for our users.

At one level, this was a sensible decision. Causing disruptions to our users might create political pressure that would stop us from taking any precautions against future spam runs from compromised local accounts. But, well, we never really found a level that our users didn't run over once in a blue moon, and when the overruns only happen once in a blue moon it's hard to iterate on tuning limits. And so things sat there from 2012 until today.

Today we had another little local spam incident (as people on Twitter might have guessed). Our ratelimit tracking code dutifully logged that this was happening and that our hypothetical ratelimits were being exceeded, and by quite a bit too:

[...] Warning: SENDER RATE LIMIT HIT: 27943.5 / 60m max 200 / [...]

So. Yeah.

It may not surprise you to hear that now we have some active ratelimits; in fact we more or less simply made our previous tracking ratelimits into enforcing ones. They're undoubtedly not perfect ratelimits, and in fact I'm fairly sure that within six months someone here sending out an entirely legitimate burst of email will run into them. But as usual the perfect is the enemy of the good. Our quest to deploy only perfect anti-spam precautions that would never inconvenience our users turned out to result in us deploying almost no anti-spam precautions, with regrettable results.

(Nor did we avoid inconveniencing users, since some of them had email bounce due to the machine in question temporarily picking up a bad sender reputation.)

We don't want to deploy significantly imperfect anti-spam precautions, for obvious reasons. Something that gets in the way of our users on a frequent basis is no good. But I've come around to the view that we need to be more willing to deploy things that are a bit imperfect and then sort out the problems when they happen. Otherwise, well, we may be looking at something like this happening all over again.

(One of those may be some sort of scanning of our outgoing email, or at least some of it. Despite my historical reservations, I now think it's possible to do this in a good way and I think that the risks of false positives may be one of those 'a bit imperfect' things we can live with, at least initially. But right now I'm kind of thinking out loud in the immediate aftermath of an incident, which gives me some biases.)