Wandering Thoughts

Plan for manual emergency blocks for your overall mail system

Last year, I wrote about how your overall anti-spam system should have manual emergency blocks. At the time I was only thinking about incoming spam, but after some recent experiences here, let me extend that and say that all entry points into your overall mail system should have emergency manual blocks. This isn't just about spam or bad mail from the outside, or preventing outgoing spam, although those are important things. It's also because sometimes systems just freak out and explode, and when this happens your mail system can get deluged as a result. Perhaps a monitoring system starts screaming in email, sending thousands of messages over a short span of time. Perhaps someone notices that a server isn't running a mailer and starts it, only to unleash several months worth of queued email alerts from said server. Perhaps some outside website's notification system malfunctions and deluges some of your users (or many of them) with thousands of messages.

(There are even innocent cases. Email between some active upstream email source (GMail, your organization's central email system, etc) and your systems might have clogged up, and now that the clog has been cleared the upstream is trying to unload all of that queued email on you as fast as it can. You may want some mechanisms in place to let you slow down that incoming flood once you notice it.)

We now have an initial set of blocks, but I'm not convinced that they're exactly what you should have; our current blocks are partly a reaction to the specific incidents that happened to us and partly guesswork about what we might want in the future. Since anticipating the exact form of future explosions is somewhat challenging, our guesswork is probably going to be incomplete and imperfect. Still, it beats nothing and there's value in being able to stop a repeat incident.

(Our view is that we've built are some reasonably workable but crude tools for emergency use, tools that will probably require on the spot adjustment if and when we have to turn them on. We haven't tried to build reliable, always-on mechanisms similar to our anti-spam internal ratelimits.)

We have a reasonably complicated mail system with multiple machines running MTAs; there's our inbound MX gateway, two submission servers, a central mail processing server, and some other bits and pieces. One of the non-technical things we've done in the fallout from the recent incidents is to collect in one spot the information about what you can do on each of them to block email in various ways. Hopefully we will keep this document updated in the future, too.

(You may laugh, but previously the information was so dispersed that I actually forgot that one blocking mechanism already existed until I started doing the research to write all of this up in one place. This can happen naturally if you develop things piecemeal over time, as we did.)

PS: Looking back over past entries I've written in this area makes me feel rueful. As has happened before, I wrote down all sorts of good ideas that I never got around to actually implementing.

Sidebar: Emergency tools versus routine mechanisms

Some people would take this as a sign that we should have always-on mechanisms (such as ratelimits) that are designed to automatically keep our mail system from being overwhelmed no matter what happens. My own view is that designing such mechanisms can be pretty hard unless you're willing to accept ones that are set so low that they have a real impact in normal operation if you experience a temporary surge.

Actually, not necessarily (now that I really think about it). It is in our environment, but that's due to the multi-machine nature of our environment combined with some of our design priorities and probably some missing features in Exim. But that's another entry.

A 'null MX' is also useful for blocking forged senders from non-email domains

When I first considered the use of a 'null MX', I was only thinking of it as a way of blocking email to hosts that don't get email (I had a special case that made some dedicated spammer behavior unusually irritating). However, there is another useful case, and that's domains that don't send email but do get forged on spam.

A while back I wrote about a persistent phish spammer that consistently sends email using the forged sender email address of 'codewizard@approject.com'. As it happens approject.com seems to be a parked domain, with a 'this domain may be for sale' website and nothing else visible. If this is true, the owner of approject.com could cut off much of this forgery by publishing a suitable 'null MX' record in their DNS (especially now that it's an official standard, as I found out when doing research for this entry). Other owners of other parked domains could similarly cut off spam being forged in their names, and frankly there's a lot of it; spammers seem to love forging email as from domains like 'confirmation.com', 'verification.net', 'system.com', and so on.

(Some of those are not parked domains, mind you.)

Even without the new(-ish) null MX RFC, you can sort of get there today for some sites through a suitable DMARC policy and SPF records, but I think that probably requires more DNS fiddling than a simple 'MX .' entry. Plus, it only applies to people who actually use DMARC or SPF to reject message, which is not that many people right now (partly because turning on DMARC or especially SPF rejection has various often unpleasant side effects). The good news is that using DMARC probably will insure that GMail and a few other big places will reject the spammer email that is claiming to be from you.

(The more DNS fiddling is required, especially the more fiddling that must contain the domain name or the like, the less likely it is that owners of parked domains and similar things will go to the bother. One attraction of 'MX .' is that it's completely generic.)

I don't know why this use for a null MX standard didn't occur to me back then. Probably I was too close to my specific little issue and not thinking generally. Spammers have certainly been abusing generic-word domains for advance fee fraud and phish spams for years.

We now have an officially standardized 'null MX' record

Years ago, I wrote about how I wished for an official 'null MX' standard so that I could clearly advertise that some of my hosts should never be sent email although they had an A record and there was a mailer listening on that IP address. In the process of writing another entry on this, I decided to look up the current state of the draft RFC from 2013. Imagine my pleased surprise to find RFC 7505: A "Null MX" No Service Resource Record for Domains That Accept No Mail.

RFC 7505 was issued June 2015, which gives me some time to have not noticed it. The official standard is a 0-preference MX to '.' (the zero-length DNS label), which is probably slightly stricter than previous interpretations but is also probably what people have been doing anyways. Since this essentially standardizes existing practice, at least some mailers have been implementing RFC 7505 since the moment it was published; others undoubtedly don't support it yet and will either fail the mail message with an unclear error, ignore the MX entry, or consider it a temporary DNS error.

Postfix apparently picked up official support for RFC 7505 in version 3.0 (released February 2015, while the RFC was in draft). I can't find any particular indication if other mailers have picked up explicit support for it (somewhat to my surprise); perhaps the authors of them are as unaware of RFC 7505 as I've been. Alternately, they were already rejecting email for things when there was only a 'MX .' MX entry, so they don't really have anything to do.

(And of course there are plenty of really old mailers out there on the Internet that will probably skip over a 'MX .' entry as clearly malformed and carry on to try to deliver the message to the IP in the A record, just the same as before.)

Since this is now an official RFC, I'm actively tempted to publish some 'MX .' entries for some hosts and see if anything happens as a result. It would be nice to think that spam senders will notice and I'll see a drop off of delivery attempts, but I'm not really that optimistic.

A single .jar recognized as several types of malware at once

In the spirit of the single email message with a lot of malware, I'll once again show you the log messages first:

1cwivp-0006vh-1M attachment application/zip; MIME file ext: .zip; zip exts: .jar; inner zip exts: .ai .b .box .class[35] .download .drive .mf .ph
rejected 1cwivp-0006vh-1M from 74.208.156.24/holmes@gmail.com to <redacted>: identified virus: CXmail/JarZip-A, CXmail/Java-A, Java/Adwind-KU

Here we have a .jar inside a .zip (which is somewhat but not totally suspicious), and from this single incoming email our system felt it found three bad things.

Sophos's detailed information for CXMail/JarZip-A is not really detailed. It's possible that this is simply their name for some apparently recognizable family of .jar-in-.zip malware; as I'd hope, some testing has shown that it's not as comprehensive as 'all .jars inside .zips'. CXmail/Java-A has similarly generic information available. Java/Adwind-KU is apparently the more well known thing, and has apparently been around for some time.

It turns out that we've seen Java/Adwind-KU before, and in the recent past cases our Sophos PureMessage reported it as 'CXmail/JarAd-G, Java/Adwind-KU'. These cases appear to have been straightforward .jar attachments. We have some earlier hits that were reported as Java/Adwind-KU alone, and back then they were were .jar-in-.zips again. All of which goes to show that this sort of stuff evolves, both in form and in recognition.

When I started writing up this case I wondered if I had a situation where several pieces of malware had all rolled themselves into a single .jar file. Now that I've looked at this it appears that this is instead a single piece of malware that triggers multiple detection signatures inside Sophos PureMessage, presumably based on how it's decided to pack itself up.

The message was sent early Saturday morning from 74.208.156.24, which isn't listed in any major DNS blocklist as I write this (it's in Barracuda's blocklist, but that's still a relatively hair-trigger one). Given its Subject, From, and To, it's obviously bad, although it didn't seem to score as spam as well as something with a virus.

(As a hint for anyone writing virus messages, if you give a message the subject of 'URGENT NEW ORDER PO1605MP1-00077' and then have the To: be the same as the From:, things are going to look more than a little bit suspicious to anyone who actually reads the message.)

PS: I don't know what .download and .drive extensions are likely to be in .jars, but they at least sound a bit suspicious. On the other hand they could be used for something completely different in real JARs; I have very little idea what Java file extensions are normally found in them. Perhaps we should figure that out so we can identify highly suspicious extensions, but that's too much work for now.

(One of the rules of anti-spam work is that there's always something more you could be doing, and thus you always have to draw the line somewhere and say 'we could do that, but let's not'.)

Spammers probably aren't paying any particular attention to you

As I sort of mentioned in yesterday's entry, I have historically written SMTP time rejection messages and other things with an eye towards denying spammers information about exactly why their attempts were rejected. This certainly looks like a perfectly rational decision; if we leak (detailed) information about rejection reasons, we give spammers a head start on working out what about their attempts needs to change in order to get their spam through. And indeed you can find plenty of large sites, like GMail and Yahoo, that absolutely refuse to give out any detailed information about rejections for this stated reason.

There's a difference here, though; we're not Yahoo or GMail. We don't have millions of users that spammers really want to send spam to; we have a thousand or so. The payoff for working around GMail's spam filtering is very high; the payoff for working around ours is extremely low. As a result, the odds that any spammers are actually paying attention to our SMTP rejection messages is, well, very low. In practice it's extremely likely that most spammers never even see them and have no interest in attempting to work around our specific tricks.

(I suspect that there are still some spammers who are paying more attention, such as people doing targeted phish spam runs and the conference spammers. Both of these groups are definitely at least somewhat targeted, and the most precise and alarming of the phish spammers are at least doing a reasonable amount of specific research on us.)

Given this realization, I've come around to feeling that your spam rejection messages might as well be reasonably informative (unless you're a big target for some reason). Maybe once in a while a spammer will read one and get a leg up, but in practice they're far more likely to be read by someone's dealing with a false positive or some other similar problem (such as trying to send an attachment type that we block). We might as well be reasonably helpful to those people, especially since some of the time they may be us (as we try to diagnose why a rejection happened).

This has probably always been the case, but I also think that when you're actively trying to block spam it's easy to get into a mindset where, to put it one way, everything is personal. Clearly the spammers are out to get their spam past you in particular and so you'd better be careful, just like the big people are. It's humbling to think that our small mail environment is generally insignificant from the spammers' perspective.

Making your SMTP rejection messages be useful for you

Our external mail gateway will reject (some) incoming messages during the SMTP conversation if our anti-spam system thinks they have too high a spam score. Until today, they were rejected with a deliberately bland and uninformative SMTP error message:

550 Rejected: this message looks too much like spam

When I designed this message, I wrote a comment about it saying 'rejections for spam deliberately give the sender an uninformative message because I don't feel like giving spammers clues'. Then today we got called in to help troubleshoot an issue where a (valid) email message from outside had bounced, and all we had to go on was this message.

Well, you know what: spammers probably aren't reading our SMTP rejection messages anyways, but we certainly do every so often. If we're reading the message this version is exceedingly unhelpful; in fact it's so generic that it's not immediately clear if it's from our system or some other system. So now our SMTP time rejection message for spam says this:

550 Rejected: CSLab PMX spam score too high (milter id <something>)

This new form does several things. First, it clearly identifies to us that the message comes from our external mail gateway. Then, between the 'milter id' and the 'PMX spam score' wording, it tells us which SMTP-time rejection is being triggered here; it's our milter-based system. Finally, the <something> is the Exim (log) ID that was assigned to the proto-message as it was being received. Using this ID we can efficiently retrieve all of the other information about the message from our logs, including the specifics of its spam score (such as they are, given that Sophos PureMessage's spam scoring is basically a black box).

Having done this exercise for one SMTP rejection message, I'm sort of tempted to do it for others. If I start from the premise that someday a user will turn up saying 'someone trying to mail me got this message', what do I want to see in the message so we can explain the situation to people?

(The good news is that I took a quick look and almost all of our other SMTP rejection messages seem to include the crucial information. For example, our 'rejected because the sending IP is in Spamhaus' SMTP rejection message actually includes the IP address, so we don't have to try to correlate logs with whatever vague information we have about the rough time the message was sent to the particular user in order to find it.)

By the way, one consideration here is that you don't necessarily want these messages to be too long, because some SMTP senders will truncate your rejection message when they report it to users (or at least they used to). I believe I've seen ones that only report the first line, for example. This is why our current rejection message is going to be relatively cryptic to anyone but us; I cautiously squeezed it down to something that I felt had a relatively high chance of making it back to us intact.

I don't get many bounce messages these days, so it's possible that modern mail systems no longer suffer from this issue. Certainly mail providers like Google and Yahoo generate quite long and verbose multi-line SMTP rejections and temporary failures. Perhaps I should add a second line with a clear, normal person focused explanation for anyone who trips over this as a legitimate false positive.

Some DNSBL developments I've just heard about

I mentioned recently that choosing DNS blocklists isn't necessarily a one-time thing that you set and forget. I always knew this in a vague and general way, but I had mostly ignored it until recently. More specifically, until I was writing that entry and wound up looking at the CBL front page, which had a March 24th announcement of news about the PSKY DNS blocklist. To wit, that PSKY had apparently been 'borrowing' Spamhaus data without authorization, that this has been stopped, and that it wasn't clear if they listed anything much any more. We've never deployed PSKY on our main mail server, but I had deployed it on my personal sinkhole spamtrap and it had been having a pretty good hit ratio. 'Had' being the operative word, because starting around the appropriate time I'd not really logged any hits against it.

All of this sent me reading through the rest of the 'Other DNSBLs' portion of the CBL's FAQ. Some of their current opinions match mine (such as Barracuda's public DNSBL being quite aggressive), but others were a surprise to me. Most prominently, the CBL people feel that the current Spamcop BL is now sufficiently safe to use as a general DNS blocklist, where my past experience with it (from several years ago) was that it was too hair-trigger. The rest of the FAQ is interesting in its own way, mostly in that it seems to confirm that there aren't really very many effective DNSBLs any more. Or at least not very many that the CBL feels that they need to talk about.

All we use in our spam filtering is Spamhaus, and I don't think there's much chance that we'll change that. The Spamhaus ZEN is as close as we can get to a high trust, fire and forget DNS blocklist, and even then our users have to opt in to it. But it doesn't hurt to keep an eye on the DNS blocklist landscape every so often (even if there seems to be less landscape than there used to be).

(That diminishing landscape is one reason I'm saddened by the news about PSKY's blocklist. When I first heard of them, they were the first new and effective DNSBL for some time, and frankly we can always do with more good spam-blocking.)

The Spamhaus CSS includes more than dedicated spam ranges

When it started out, the Spamhaus CSS was primarily there to list IP addresses and address ranges used for snowshoe spamming (this was explicitly covered in the announcement of the Spamhaus CSS, and even helped give the CSS its name). However, things have changed somewhat since 2009, both with Spamhaus and perhaps with snowshoe spammers themselves. Specifically, the CSS is now described as:

The Spamhaus CSS list is an automatically produced dataset of IP addresses that are involved in sending low-reputation email. CSS mostly targets static spam emitters that are not covered in the PBL or XBL, such as snowshoe spam operations, but may also include other senders that display a risk to our users, such as compromised hosts.

(The italics are mine.)

Pragmatically, my observations say that it's not a 'may' here, it's a 'definitely does'. On my sinkhole SMTP server, most or almost all of the SBL CSS hits that I see these days are also in the CBL and/or the PBL (based on looking at recent hits). There was a day when many of the SBL CSS hits were dedicated snowshoe spam areas, but that day is evidently over. Either the snowshoe spammers are now sending their spam through compromised IPs as well as their own dedicated ranges, or the characteristics of genuine snowshoe spam and the sort of spam you get via compromised IPs are merging so that Spamhaus now has given up on telling them apart.

(I suspect that at least the first is definitely true, although it's a bit odd that careful snowshoe spammers would be willing to rent IPs that are already known as compromised, or worse are outright listed on the PBL as 'should never accept email from this IP'. You'd think that that would be asking for delivery problems and people wouldn't really want to pay for such IPs. Maybe access to them goes for really cheap.)

This isn't particularly a bad change, but it does have implications for what DNS blocklists you may want to check or how you may want to report things. Personally I'm most interested in knowing real snowshoe spam IPs and IP ranges, so I now want to check the SBL only after checking the CBL and the PBL. And if you have more elaborate reporting capabilities for DNSBL hits, you definitely want to check and report all data values returned from, say, the Spamhaus ZEN, so that you can see that a snowshoe IP was also in the PBL or the CBL or both.

Also, all of this goes to show that choosing and setting up DNS blocklists is not necessarily a one-time thing. If you have anything except conservative settings, it's something that requires a certain amount of active ongoing maintenance, both to look at your results and to keep up on news about DNS blocklists.

(Just using the Spamhaus ZEN and not caring too much about why things are blocked is conservative. Spamhaus is unlikely to do anything weird and changes like this don't affect you unless you care about specifically why something is blocked.)

An odd and persistent year old phish spammer

We have a number of more or less internal mailing lists for things like mailing all of the technical staff. They have at least somewhat unusual names and don't appear in things like email directories or most users' address books. Back almost a year ago (21st April 2016), one of them got a phish spam:

From codewizard@approject.com [...]
Received: from [177.47.160.250] (helo=approject.com) [...]
From: "Capital One 360" <codewizard@approject.com>
Subject: Your Capital one 360 Account Urgent Login Reminder

LOOK FOR THE ATTACHED FILE AND OPEN

(With an attached PDF.)

Slightly over a month later, the same address got another one:

From codewizard@approject.com [...]
Received: from [95.213.155.178] (helo=approject.com) [...]
From: "USAA SECURITY" <codewizard@approject.com>
Subject: Your Account Log-on Reminder

A week later it got a third one, with the same MAIL FROM (and EHLO), but from a different IP address yet again. Then a fourth two weeks later.

At this point I'd had enough, so I threw the MAIL FROM of codewizard@approject.com into the per-address server side email blocks for this particular address. You can probably guess what has happened periodically ever since then:

2017-03-23 18:11:31 H=(approject.com) [46.39.225.151] F=<codewizard@approject.com> rejected RCPT <redacted>: blocked by personal senders blacklist.

(As I write this, that IP address is on the Spamhaus CSS.)

It's clear that whatever is doing this spamming is widely dispersed, very persistent, and is using a basically unique address list that it has a death grip on (this internal mailing list of ours hasn't started getting other sorts of spam, just this one phish spammer). Maybe this is wandering malware that is now operating more or less autonomously (like some do), or maybe this is someone running a long-term campaign who cannot be bothered to disguise the distinctive signatures here (those being the envelope sender and the EHLO).

(This isn't the first time I've seen spammer persistence illustrated, but I think it's the first time it's clearly a single spammer or spam agent instead of address lists being shared and reshared endlessly.)

PS: Since various aspects of this phish spam have apparently mutated over time, it's probably not autonomous malware in action but instead someone running a long-term campaign. I don't know why they're so fixated on using this very distinctive MAIL FROM, but it's certainly handy so please don't change, whoever you are.

Malware is sometimes sent through organized, purchased infrastructure

Every so often, I wonder where malware comes from. Well, in a mechanical sense; does it come from infected machines, or from rented botnets, or what? Today we got a malware attack campaign that gave us a very clear answer: it came from dedicated, custom-built infrastructure.

Between 12:17 and 12:28 today, a cluster of four IP addresses tried to send us 376 email messages. All of them had a HELO and verified host name of confidentialdocumentdelivery.com, and the MAIL FROM was 'service-<tagged-address>@confidentialdocumentdelivery.com'. All of them that did not get rejected for other reasons had a .doc file that Sophos identifies as CXmail/OleDl-V. To make things look more tempting, they all had some common mail headers:

To: <whoever>
Subject: Confidential Documents Delivery
From: "Document Delivery" <service@confidentialdocumentdelivery.com>

(They also appear to have had valid DKIM signatures, just in case you think DKIM signatures on email are any sign of trust.)

At the time, confidentialdocumentdelivery.com was (and is) in the Spamhaus DBL, and all four IPs involved were in the Spamhaus CSS, probably among other blocklists. The four IPs in question are all in AS202053 (or), 'UpCloud Cloud Servers' according to RIPE information. Their DNS PTR records at the time were all 'confidentialdocumentdelivery.com', but they've since been recycled to other PTRs. The domain itself seems to have been registered only today, assuming I believe the whois data.

All of this makes it clear that these weren't infected machines, hijacked machines, or a rented botnet. This was a whole set of carefully built infrastructure; someone figured out and bought a good domain name, rented some VPSes, assigned DNS, configured a whole set of mail sending infrastructure (complete with VERP), and used all of this to deliberately send out malware, probably in large bulk. This was an entire organized campaign on dedicated infrastructure that was put together for this specific purpose.

(The infrastructure may or may not have been custom built. For all I know, there are people who sell spammers the service of 'I will set up your sending infrastructure; you provide the domain name and some VPSes and so on'. And if it was custom built, I suspect that the malware gang responsible for this will reuse much of the software configurations and so on for another malware barrage.)

The thing that puzzles me is why you would go through all of the effort to plan and develop this, execute the plan at good speed and with solid organization (if the domain was only registered today), and yet use malware that Sophos and presumably other could already recognize. According to Sophos's page, recognized versions of this have been around since January, which I suspect is an eternity in terms of malware recognition.

(For the curious, the four IPs are 94.237.24.77, 94.237.30.162, 94.237.30.163, and 94.237.30.164. Out of those two /24s, 94.237.30.112 and 94.237.30.153 are also currently on the Spamhaus CSS.)