Wandering Thoughts archives

One anti-spam thing I like is per-person (or per-address) blocklists

I've come to feel that one of the powerful anti-spam things that you can do in any environment with a shared mail system is to support some form of individual filtering and blocklists at the SMTP level. Given the SMTP DATA error problem, the conceptually easy options to support here are blocklists based on the sending host and on the MAIL FROM envelope address, since those are easily done at RCPT TO time on a per-recipient basis.

The general case for supporting individual filtering is pretty straightforward. One size does not fit all, since both people's email patterns and their level of caution (and tolerance for spam) vary. Individual blocking empowers people to block things for themselves that you could never get permission or agreement to block on a global basis. In turn this is likely to make them happier with your email system, partly because they will be getting less spam and partly because they'll probably feel more in control of the whole process.

Blocking at SMTP time is harder than the alternatives, especially on a per-user basis, but it's doable. I advocate for doing it despite the difficulty partly because SMTP-time rejection has various technical advantages, partly because I plain like it, and partly because I feel that people in general are likely to be more comfortable with filtering that returns error messages to the sender in cases of false positives, which today requires rejecting at SMTP time.

(Perhaps the last is projecting my feelings on to other people, as we certainly have a fair number of people who automatically discard incoming spam without appearing to ever worry about it.)

A per-address blocklist feature is convenient even on a mail server that's only used by you, because it provides a nice way to start closing down single-purpose addresses when they start getting spammed or abused.

(If you run your own mail server, you really should set up some sort way of having controllable single-purpose addresses.)

A confession: I find rejecting spam at SMTP time to be more satisfying

In a modern mail environment, there are a number of places where you can block or drop spam. Of these places, in some ways the hardest place to support is rejecting things during the SMTP conversation. It may take gory hacks to get your inbound mailer to do spam checks during the conversation, there's the SMTP DATA rejection problem, and various other problems.

In the past I've written on various good technical reasons to reject at SMTP time, and I certainly (still) believe all of them. Rejecting spam at SMTP time reduces the risk of false positives and will often (although not always) reduce the load on your mail system, perhaps substantially, and so on. But I have to confess that there is an additional reason that I really like SMTP time rejection of spam: I simply find it a lot more satisfying.

However irrational it is, it simply feels better to reject a spam message at SMTP time than to let it on to my systems. Rejecting at SMTP time feels like I am in some tiny little way giving the spammer the middle finger and throwing their own spam back at them. And on the other side, accepting their spam and then throwing it away later makes me feel just a little bit slimed by that spam, even if all steps are automated and the end result is exactly the same. It even makes me feel a bit irritated to accept and then dump spam.

(Part of it is that with SMTP rejections we are visibly registering some sort of objection to the mail, but that's not all of it by any means.)

I'm aware that this is an irrational thing, but rejection at SMTP time just makes me happier. As a result, I'm willing to go out of my way to create mailer configurations and so on that enable us to do this (and enable us to offer this to users as something they can enable).

(I suspect that part of why I find SMTP time rejection to be more satisfying is that it is the right place to do it at a technical level. Accept then drop is clearly technically inferior to rejecting during the SMTP conversation.)

There are three places spam filtering can happen these days

These days, there are three different places where (anti-)spam filtering can happen in an email system:

1. In the user's mail client, assuming that you let them run one that has filtering rules (and most of them do). It's worth noting that there is effectively no client side filtering in webmail systems.

2. On the server once the email has been accepted at the SMTP level. This is the domain of people running procmail from their .forwards, as well as various sorts of system-wide filtering.

3. At SMTP time, so that undesired messages get SMTP level rejections. This is easily done for things like DNS blocklist checks, but can often be done with content filters as well.

Client side filtering is essentially entirely in the hands of users to both set up and maintain. You can try to distribute some standard filtering rules if you happen to have standard clients that you support, but I'm not sure how much support there is for updating them later except through manual user action (which we all know is very unlikely to happen). On the other hand, client side filtering fully empowers users to do whatever filtering they want with whatever tools they can put together. It also requires the least effort on your part, since you basically don't have to do anything; MUA authors maintain the filtering system and users maintain whatever filtering rules they want.

As someone who doesn't like to quietly drop email, it's my personal view that SMTP time rejection is the best option if you can manage it. Early SMTP rejection before DATA can significantly reduce the load on your mail system, but rejection after DATA may help less (since you have to receive the message and then run it through potentially expensive checking). Unfortunately the limitations of SMTP may make DATA-time rejection potentially chancy, since they apply to all recipients.

On the server after the mail has been accepted from the sender has been the traditional place to do filtering, at least on Unix machines. It's generally been much easier to run email through additional steps here rather than at SMTP time and there have long been robust tools for filtering, both at the individual user level and at the overall MTA level. Procmail is a classic even if it's not all that user friendly.

(As mentioned, people using webmail generally don't have a client side, so for them it is some form of server-side filtering or nothing. This filtering may run when messages are received or just when people log in to their webmail and start doing things.)

Only at SMTP time can you genuinely reject the email (although the sender may or may not pay any attention to your rejection; spammers probably won't). After that, about all you can really do is throw the message away (see this for details on why). On the other hand, if you want to let users go through their filtered out email you basically have to accept it first.

Red Hat has really doubled down on being email spammers

Back in June, I wrote about how we'd gotten marketing email from Red Hat to a RHN contact address. At the time I wrote an angry complaint email to Red Hat about it, from my core email address. This is the only time I have given Red Hat my core email address; it is not, for example, the address I use on their Bugzilla.

On November 10th, I got the following email message from Red Hat, once again from their very own SMTP server:

From noreply@engage.redhat.com Tue Nov 10 [...]
Received: from mail01.engage.redhat.com ([204.92.22.83]) [...]
[...]
From: "Red Hat" <email@engage.redhat.com>
Subject: Win a complimentary pass to RSA Conference in Nov. 18 virtual event

If you're looking for a reason to register for Secure Foundations for Today and Tomorrow | A Red Hat virtual event, look no further. In the virtual event on November 18, you'll hear from Red Hat executives, technical experts, and IKEA, and learn more about: [...]

There are no polite words for a company that takes email addresses that have sent abuse complaints to them and recycles those email addresses into their mailing lists as fresh addresses. Such a company is a spammer, period, full stop.

So: Red Hat are email spammers.

Sic transit gloria mundi, and all that. There was a time when I would have believed that Red Hat would never do this, that it had far too many principles to even get close to this. Clearly I was wrong, and I guess I found out how far Red Hat was willing to go (as I wondered in my initial entry).

I sent another email complaint, because somehow I am still an optimist. It has been almost a month and just as the first time, I have had exactly no response from Red Hat. Dead silence to abuse complaints is of course another sign of a spammer, so I shouldn't be surprised.

(Sending out your email with a 'noreply' address is of another one of those signs. Real mailing list software uses envelope senders to track bounces and handle them; software that is never going to remove email addresses when they bounce obviously has no need for such niceties.)

I haven't gotten any further marketing spam messages from Red Hat, but I expect it's only a matter of time (and thus I expect to someday write a third entry to confirm that it's happened). If they continue it long enough, 2017 will be an interesting year.

Sidebar: Who engage.redhat.com appears to be

While various pieces of information (such as good reverse DNS) ties this to Red Hat with Red Hat's active cooperation, Red Hat appears to have outsourced part of their spamming to something called eloqua.net, aka en25.com, aka eloqua.com. To quote the latter's website, this is 'Oracle's B2B Cross-Channel Marketing solution'. There is some moderate irony in that, but I suppose that for both Oracle and Red Hat, money is money.

Outlook.com now has collected some SBL listings

I mentioned on Twitter that portions of outlook.com are now on the SBL. At the moment there are two listings for protection.outlook.com hosts; SBL272953 from October 11th and SBL273948 from October 21st. Both spam samples quotes by Spamhaus show the signs of a null sender, so clearly these people are as entrenched as I thought. Microsoft also has a Hotmail SBL listing, SBL268930, from September 10th.

(All Microsoft SBL listings can be found here, which is a link I want to keep for my own reference if nothing else.)

Clearly Microsoft doesn't care enough about these SBL listings to do anything about them. It's not clear why this is so, though. Perhaps the Microsoft abuse system is undermanned and overwhelmed. Perhaps a SBL listing doesn't affect delivery to enough places for Microsoft to care (especially a SBL listing for just one or two IPs out of the many that protection.outlook.com hosts use). Perhaps Microsoft simply hasn't noticed the SBL listings.

Locally we've seen connections from one of these IPs in the past week, and all of the deliveries were for null sender address so they were almost certainly spam. This means that I don't currently have to worry about the effects on our users of outlook.com getting more widely listed in the SBL (which is a concern, since some of the university's own email comes from there).

(Only some users subscribe to SBL-based rejection, but in the past SBL listings have clearly been a significant input to the spam score our commercial anti-spam system computes for messages. My unscientific belief is that a great many people filter their email based on that score, so widespread SBL listings for outlook.com could well push the scores for outlook.com email into 'filter away' territory. If this happened, there would be basically nothing we could do about it.)

When setting up per-thing email addresses, make sure you can turn them off

One of the reasonably good weapons in the war against spam is never giving companies your real, core email address but instead creating different individualized email addresses for each place that requires an email address from you. When email addresses inevitably leak or get abused, you can immediately finger the culprit simply by what email address the crud is coming in to. If you start getting spam email to 'you.vendor1@...', well, you know who to blame.

There are a lot of ways to do this, depending on exactly where you have your email; some email providers let you add arbitrary suffixes to your email address after a special character, for example. But however you do this, there is an obvious but important feature you should try to have if at all possible: you should be able to turn off such addresses. Generally such turned off addresses should be rejected during the SMTP transaction, although in some circumstances you might want to turn them into spamtraps instead.

The reason why is pretty straightforward. As I can say from personal experience, after a while at least some of your addresses will become so contaminated that the spam they get is no longer at all interesting. Once an address gets leaked badly enough that the advance fee spammers and the phish spammers and so on get their hands on it, well, there's no end to them.

(I don't recommend that you immerse yourself in the (anti-)spam world, not unless it's your profession. Trying to track advance fee fraud and phish spam is an endless task.)

Most systems for individualized addresses can probably already do this, but if you're building one (for yourself or for a general population), remember to include this. It may take some extra work, but you'll thank yourself in the long run.

(The simple approach is to make the address not exist any more, so it's rejected at SMTP time the same as any other nonexistent local address. The more advanced one is to still reject at SMTP time but to keep track of things like how many attempts to mail it there were recently and so on, and let people see this. Although, honestly, I'm not sure how many people will really care about that. You probably do want to keep track of old turned off addresses so that people get a warning if they're recreating them by accident.)

Some theories about what spammers get out of using null sender addresses

In light of spammers exploiting outlook.com with null sender addresses, one obvious question to ask is why they bother doing this. Well, that's not quite the right question, because the obvious answer is 'it helps their spamming somehow'. My real question is how it helps, and on this I have a few theories.

It's possible that anti-spam systems are more likely to let email from null sender email addresses through than other email, but I don't really believe this. Outlook.com is not the only place where spammers can use null sender addresses, and so if it was useful this way I'd expect spammers to have been using null senders already. Instead I believe that outlook.com spam is pretty much the first big source of this that I've seen.

There's a number of possible things this might do on outlook.com specifically. First, perhaps the spammers have figured out how to exploit a message submission path that requires less authentication, does less spam checking on outgoing email, or is easier to use in bulk but that requires null senders. Here, the null sender is more or less a side effect of the submission path and the spammers don't particularly care about it.

An obvious speculation is that the spammers have found that using a null sender slows down outlook.com's abuse handling process. I don't particularly believe this, since the null sender spam I've trapped has plenty of peculiar internal Microsoft headers. Given that Microsoft hosts disparate things behind the outlook.com name, I'd expect that these headers are what Microsoft actually uses to backtrack spam.

However, there's a related possibility. It's quite possible that a place like outlook.com uses the volume of SMTP time rejections as a signal of badness. If a lot of the email that a particular address sends out gets rejections, well, that's probably worth paying attention to. It could well be that using a null sender mostly defeats this precaution, buying the spammers more time (and more spam) before outlook.com's automated measures stop them. Of course this shouldn't really be the case, since outlook.com has those internal tracking headers even with a null sender, but, well, it's already been established that Microsoft is falling down on the job.

Finally, there's an obvious answer: spammers are simply saving themselves the effort of coming up with sender addresses, especially ones that won't trip over SPF or DMARC or whatever policies, or hit other issues. I don't think I've ever seen a spam of this nature that wants you to reply to the sending address; when they want email replies at all, the spam has a Reply-To: to somewhere else (and if the From: matters, that can be forged). Given that outlook.com lets the spammers use the null sender, well, that gets them out of that little bit of work.

(Of course all of this is empty theorizing about something I'll probably never have the answer to. But the whole situation bugs me, as you can probably tell. And if spam from null senders is going to trend up in general, that's going to affect mail filtering systems.)

The null sender spammers now seem to be entrenched on outlook.com

A bit over a month ago I wrote about how spam from outlook.com had started showing up with a null sender address (a MAIL FROM of '<>'). It will probably not surprise you to hear that this spam has continued, and in fact has likely intensified. Based on what I've seen in our logs and in a spamtrap that I enabled in order to collect samples of this spam, a number of spammers appear to have worked out that Microsoft will let them get away with this and are happily spamming away.

(One of the spam samples I captured was a reasonably targeted phish spam, which makes me even more annoyed with Microsoft.)

Our anti-spam appliance keeps logs, of course, and this gives me a way to assess just how much null sender spam has been showing up here. Based on logs from the past ten full days, it breaks down like this:

• 490 null sender messages sent to us from .protection.outlook.com hosts, out of 2,570 messages from them in total. So about one in five.

• 249 had a 90% or higher spam score; 30 had one in the 80% range and 17 in the 70% range, which is roughly our cutoff for scoring something as spam. So more than half were spammy enough that our system saw them as clear spam.

• Out of the outlook.com messages without null senders, only 23 scored 90% or higher, 16 in the 80% band, 4 in 70%, and 3 in 60%. In fact, 1860 of the 2080 scored under 10%.

Now, this doesn't mean that our anti-spam appliance has scored these correctly either way (and in fact I suspect that almost all of the null sender messages were actually spam). But it does strongly suggest that the messages with null senders are very much skewed towards spam instead of legitimate email (and obvious spam at that), and thus that this is a signal that Microsoft should be looking at and doing something about. If they cared and paid attention, that is. Which they clearly don't.

(Someday they will, when sufficiently many spammers figure this particular trick out that the wave of spam becomes a real problem for Microsoft. But that's probably going to take a while and in the mean time Microsoft's corporate indifference is subjecting the rest of us to a steadily increasing barrage of spam from their servers.)

How many recent sender domains are in the Spamhaus DBL

The Spamhaus DBL is, well, let's quote it directly:

The Spamhaus DBL is a realtime database of domains (typically web site domains) found in spam messages. [...]

Per Spamhaus's documentation, the recommended or best way of using the DBL is to check URLs in incoming messages against it. However you can also use it to check domain names from other sources, such as DNS hostnames, EHLO claimed names, and the host or domain name in the envelope sender address (the SMTP MAIL FROM).

For reasons beyond the scope of this entry, I got curious about how many of the domains sending us email over the recent past might be (still) listed in the DBL. To get a rough idea of this, I extracted the sender domain for all accepted email on our external MX gateway for roughly the past ten days and checked them all. The headline results surprised me:

Out of 10,397 different sending domains, 1,422 were on the DBL.

This is a lot more than I expected. Note that this is a count of domains, not email volume; to put it one way, 'gmail.com' is one domain just as 'aftencia.review' is, but the former is sending us many more email messages than the latter.

Since this is email the gateway accepted, it excludes email that was rejected during the SMTP conversation for various reasons. I've noticed that there's a fairly decent correlation between SBL listed IPs and DBL listed sender domains (eg many IPs that are on the SBL CSS seem to use MAIL FROMs that are in the DBL, probably unsurprisingly).

(I'm presenting such relatively odd numbers because it's much more work to get more interesting ones, such as what percentage of accepted email messages those DBL-listed senders are responsible for. Crude shell scripts don't make what are effectively cross-table joins very easy. Also, I started out expecting a very low DBL hit rate, which would have made detailed stats fairly pointless.)

PS: While there were quite a number of new TLDs in the DBL listed domains, it turns out that the three most common TLDs were .com, .net, and .eu (followed by .download and .xyz). However, somewhat over half of the .net domains come from .in.net; if considered separate from .net, it would be the the fourth most common 'TLD' (and .net would drop out of the top five).

Spam from outlook.com has gotten worse (well, for me)

Microsoft's outlook.com has been a spam sewer for me for some time, which is not really surprising for Microsoft but still annoys me. Recently things got a bit worse and more annoying than usual, for a simple and nominally trivial reason: spammers have started sending spam through outlook.com with a null sender address (a MAIL FROM of '<>').

(The spam itself was ordinary advance fee fraud spam.)

This irritates me for several reasons. First, a null sender is an administrative MTA level thing. Microsoft has almost no reason to allow users to send email through them with it, and there are a lot of reasons to disallow it. The second issue is that many mail configurations apply less checks to null sender addresses (usually for historical reasons), so allowing people to use null sender addresses for real mail just helps spammers get their spam past checks. And third, given that outlook.com itself is a multi-tenant thing (as I've found out in the past), allowing tenants to use the null sender makes it that much harder for people on the receiving end of outlook.com's spam cannon to distinguish between bad and potentially good email from outlook.com. Now we don't have even the MAIL FROM domain as a signal, because there isn't one on the null sender.

Microsoft doesn't care, of course, If Microsoft cared at all, their outlook.com operation would look rather different (and they would have a different response to abuse complaints); why, they might run outgoing email through a spam detector and then refuse to send obvious bulk advance fee fraud messages. Instead Microsoft has clearly taken the overall attitude that they're too big for people to block email from and so they'll do more or less the minimum amount of work to avoid people revolting.

(As readers may have gathered, I do not have very positive views of basically any large email provider (I'd say 'free', but I believe Microsoft charges for some email hosting they do).)