Wandering Thoughts archives

Spam victims don't care what business unit is responsible for the spam

So what happened is that the other day I got some spam that our MTA received from one of the outbound.protection.outlook.com machines. Since sometimes I'm stubborn, I actually tried reporting this to abuse@outlook.com. After some go-arounds (apparently the Outlook abuse staff don't notice email messages if they're MIME attachments), I got the following reply:

Thank you for your report. Based on the message header information you have provided, this email appears to have originated from an Office 365 or Exchange Online tenant account. To report junk mail from Office 365 tenants, please send an email to junk@office365.microsoft.com and include the junk mail as an attachment.

Ha ha, no. As I put it on Twitter, your spam victims don't care about what exact business unit is responsible for the specific systems or customers or whatever that sent spam. Sorting that out is your business, not theirs. Telling people complaining about spam to report it to someone else is a classic 'see figure one' response. What it actually means, as everyone who gets this understands, is that Microsoft doesn't actually want to get spam reports and doesn't actually want to stop spam.

Oh, sure, there's probably some internal bureaucratic excuse here. Maybe the abuse@outlook.com team is being scored on metrics like 'spam incidents processed per unit time' and 'amount of spam per unit time', and not having to count this as 'their' spam or spend time forwarding the message to other business units helps the numbers out. But this doesn't let Microsoft off the hook, because Microsoft set these metrics and allows them to stand despite predictable crappy results. If Microsoft really cared, outlook.com would not be the massive spam emitter that it is. Instead Microsoft is thoroughly in the 'see figure one' and 'we're too big for you to block' business, just like a lot of other big email providers.

(For people who do not already know this, 'see figure one' refers to a certain sort of grim humour from the early days of Usenet and possibly before then, as covered here and here. The first one may be more original, but the 'we don't care, we don't have to, we're the phone company' attitude is also authentic for how people read this sort of situation. Application to various modern organizations in your life is left as an exercise to the reader.)

Looking more deeply into some SMTP authentication probes

Back when I wrote about how fast spammers showed up to probe our new authenticated SMTP service, I said:

[...] I can see that in fact the only people who have gotten far enough to actually try to authenticate are a few of our own users. Since our authenticated SMTP service is still in testing and hasn't been advertised, I suspect that some people are using MUAs (or other software) that simply try authenticated SMTP against their IMAP server just to see if it works.

Ha ha. Silly me. Nothing as innocent as this is what's actually going on. When I looked into this in more detail, I noticed that the IP addresses making these SMTP authentication attempts were completely implausible for the users involved (unless, for example, some of them had abruptly relocated to Eastern Europe). Further, some of the time the authentication attempts were against usernames that were their full email addresses ('<user>@ourdomain') instead of just their login name. But they were all real usernames, which is what I noticed first.

What I think has to be going on is that attackers are mining actual email addresses from domains in order to determine (or guess) the logins to try for SMTP authentication probes, rather than following the SSH pattern of brute force attacks against a long laundry list of login names. This makes a lot of sense for attackers; why waste time and potentially set off alerts when you can do a more targeted attack using resources that are already widely available (namely, lists of email addresses at target domains).

(Not all of the logins that have been tried so far are currently valid ones, but all but one of them have been valid in the past or at least appear in web searches.)

So far there's only been a small volume of these probes. It's quite possible that in the long run these will turn out to be in the minority and the only reason I'm noticing them right now is that other attackers have yet to discover us and start more noisy bulk attempts.

(I suspect that at least some of the attackers are doing this because of the presence of the DNS name 'smtp.cs.toronto.edu'.)

Spammers show up fast when you open up port 25 (at least sometimes)

As part of adding authenticated SMTP to our environment, we recently opened up outside access to port 25 (and port 587) to a machine that hadn't had them exposed before. You can probably guess what happened next: it took less than five hours before spammers were trying to rattle the doorknobs to see if they could get in.

(Literally. I changed our firewall to allow outside access around 11:40 am and the first outside attack attempt showed up at 3:35 pm.)

While I don't have SMTP command logs, Exim does log enough information that I'm pretty sure that we got two sorts of spammers visiting. The first sort definitely tried to do either an outright spam run or a relay check, sending MAIL FROMs with various addresses (including things like 'postmaster@<our domain>'); all of these failed since they hadn't authenticated first. The other sort of spammer is a collection of machines that all EHLO as 'ylmf-pc', which is apparently a mass scanning system that attempts to brute force your SMTP authentication. So far there is no sign that they've succeeded on ours (or are even trying), and I don't know if they even manage to start up a TLS session (a necessary prerequisite to even being offered the chance to do SMTP authentication). These people showed up second, but not by much; their first attempt was at 4:04 pm.

(I have some indications that in fact they don't. On a machine that I do have SMTP command logs on, I see ylmf-pc people connect, EHLO, and then immediately disconnect without trying STARTTLS.)

It turns out that Exim has some logging for this (the magic log string is '... authenticator failed for ...') and using that I can see that in fact the only people who have gotten far enough to actually try to authenticate are a few of our own users. Since our authenticated SMTP service is still in testing and hasn't been advertised, I suspect that some people are using MUAs (or other software) that simply try authenticated SMTP against their IMAP server just to see if it works.

There are two factors here that may mean this isn't what you'll see if you stand up just any server on a new IP, which is that this server has existed for some time with IMAP exposed (and under a well known DNS name at that, one that people would naturally try if they were looking for people's IMAP servers). It's possible that existing IMAP servers get poked far more frequently and intently than other random IPs.

(Certainly I don't see anything like this level of activity on other machines where I have exposed SMTP ports.)

Email from generic word domains is usually advance fee fraud spam

One of the patterns I've observed in the email sent to my sinkhole SMTP server is what I'll call the 'generic word domain' one. Pretty much any email that is from an address at any generic word domain (such as 'accountant.com', 'client.com', 'online.com', or 'lawyer.com') is an advance fee fraud spam. It isn't sent from or associated with the actual servers involved in the domain (if there's anything more than a parking web page full of ads), it's just that advanced fee fraud spammers seem to really like using those domains as their MAIL FROM addresses and often (although not always) the 'From:' in their message.

Advance fee fraud spammers use other addresses, of course, and I haven't done enough of a study to see if my collection of them prefers generic nouns, other addresses (eg various free email providers), or just whatever address is attached to the account or email server they're exploiting to send out their spam. I was going to say that I'd seen only a tiny bit of phish spam that used this sort of domain name, but it turns out that a recent cluster of phish spam follows this pattern (using addresses like 'suspension@failure.com', 'product@client.com', and 'nfsid@nice.com').

I assume that advance fee fraud spammers are doing this to make their spam sound more official and real, just as they like to borrow the domains of things associated with the particular variant of the scam they're using (eg a spam from someone who claims to be a UN staff member may well be sent from a UN-related domain, or at least from something that sounds like it). I expect that the owners of most of these 'generic word' domains are just using them to collect ad revenues, not email, and so don't particularly care about the email being sent 'from' them.

(Although I did discover while researching this that 'nice.com' is a real company that may even send email on occasion, rather to my surprise. I suspect that they bought their domain name from the original squatter.)

(This elaborates on a tweet of mine, and is something that I've been noticing for many years.)

Unsurprisingly, random SMTP servers do get open relay probes

One of the things I do with my sinkhole smtp server is run a copy of it on my home machine. Unlike my office workstation, my home machine has never been an active mail machine; it has nothing pointing to it and no history of various (pseudo)email addresses that attract spam. Under normal circumstances there should be absolutely no one with any reason to connect to it.

Indeed, it doesn't attempts to send me any email (spammers might plausibly try, say, postmaster@<machine>). What it does get is a certain amount of open relay probes. Originally these probes were sent with outside MAIL FROM:s (and outside RCPT TOs, obviously), but lately they've been forged to come from various addresses at the machine's overall domain.

(What's actually pretty interesting about this is that the overall domain isn't valid for email; it has neither an A nor an MX entry, and never has. The spammers are just assuming that, eg, 'support@<domain>' is a valid address and using it as the MAIL FROM.)

It used to be that the relay probes made one or two attempts and then stopped. The recent run of relay probes has dumped a whole series of email on my machine all at once, varying at most the MAIL FROM address; I assume it's trying to see if some will go through where others fail. At the moment addresses on GMail appear to be the popular collection point for results. The Subject lines of recent relay attempts clearly contain tracing information and suggest that the software involved is normally used against things that require SMTP AUTH, as it seems to be including passwords in the Subject: information.

The exact details and mechanisms have changed from earlier attempts and will undoubtedly change again in the future. What's really interesting is two things: people really do scan more or less random addresses in an attempt to find open SMTP relays, and when they find something they don't immediately start trying to shovel spam through it but instead attempt to verify that it actually is open.

(Some days I'm tempted to manually 'relay' one of these messages to its collection point just to see if there would be a future attempt to spam through my machine. But so far that's far too much work and probably a certain amount of risk.)

Our current email anti-virus system is probably ineffective now

Last month I noticed that classical viruses by email were still around, despite a past history of low virus detection by our main mail system. Well, funny you should mention that. As it happens, late last week the whole university was battered a large tide of infected phish/virus emails over several days (and we had several infections ourselves). If our anti-spam system is any good at detecting viruses, I'd expect a serious uptick in virus detection because the actual rate of virus emails was clearly up significantly.

The good news is that there is a definite uptick over the two days with the bulk of the attack. The bad news is that it is not to very high numbers; 81 Monday, 95 Tuesday, 112 Wednesday, 101 Thursday, and 47 Friday. A normal weekday appears to run around 50 viruses detected a day. And it's highly likely that at least some viruses made it through this screening to reach our users.

(Note that some of these 'viruses' are actually phish spam. It's possible that they're phish spam with executables attached; I don't know.)

It's possible that some of the viruses were detected as spam, but there are two strikes against this. The first is that detected spam volume does not seem to fluctuate much over those days. The second is that detecting viruses as spam instead is actually bad for us; if it's detected as an actual virus, the anti-spam system removes the viral content instead of merely marking the Subject: line.

Unfortunately I don't know what options we have, and also how much work it's worth putting into this in general. After all, if our actual virus email rate is quite low outside of anomalies such as this it probably doesn't matter that our current anti-spam system seems at best so-so at detecting viruses. We could plow a lot of time and effort into evaluating (free) options like ClamAV only to find out blocking only a small extra amount of email, which hardly seems worth it.

(I have complicated attitudes on anti-virus stuff, but the short summary is that I think it's very dangerous to put much emphasis on email filtering keeping them out.)

I've now seen comment spam attempts from Tor exit nodes

As I mentioned on Twitter, I've recently started seeing some amount of comment spam attempts from IPs that are more or less explicitly labeled as Tor exit nodes. While I haven't paid exhaustive attention to comment spam sources over time, to the best of my awareness this is relatively new behavior on the part of my comment spammers. To date not very many comment spam attempts have been made from Tor IPs and other sources still dominate.

Since none of the comment spam attempts have succeeded, I face no temptation to block the Tor exit nodes. There are plenty of legitimate uses for Tor and I'd much rather have my logs be a little bit noisier with more failed comment spam attempts than even block a legitimate anonymous comment.

(Really I only block comment spam sources because I'm irritated at them, not because I think they represent any particular danger of succeeding. So far I've seen no sign that the robotic form stuffers are changing their behavior in any way; they've been failing for more than half a decade and I expect them to keep failing for at least the next half a decade. It's very unlikely that my little corner of the web is important enough to attract actual human programming attention.)

Given that this is a recent change, my suspicion is that Tor has simply become increasingly visible and well known to spammers through its appearance in stories about Silk Road and other hidden services (and people using it). Apparently some malware is now starting to use Tor to contact its command and control infrastructure, too, and certainly we've seen attackers use Tor to hide their IP origin when they access cracked accounts.

(Ironically this makes access from Tor exit nodes a glaring sign of a cracked account for us, since basically none of our users do this normally. Conveniently there are sources for lists of Tor exit nodes (also).)

Somewhat to my surprise, classical viruses by email are still a thing

Normally, my sinkhole spam-capturing SMTP server is set up so that it rejects as much as possible of what I consider boring spam. The other day I decided to run it completely unfiltered for a while, accepting everything no matter how obviously bad it was or whether it was going to an address that had ever existed. Already something interesting to me has turned up in the results.

In statistics drawn from our production mail system, I've previously noticed that viruses in email are way, way down. Much to my surprise, the first day of operating my sinkhole server completely unfiltered got me no less than five classical virus-laden email messages (out of 60 messages received so far). And when I say they're classical, they're really classical:

• all are Windows executables, one straight as a .pif file and four inside .jpg.zip files (the one that I extracted was a .jpg.exe file).

• all appear to have come directly from end-user machines, not relayed through anyone's mail systems (based partly on DNS PTRs associated with them, network areas, etc). Three out of the four IPs involved are listed in the PBL.

• all four IPs involved are currently listed in the CBL.

Four of the five arrived in one burst and are all the same zipfile and executable; although they came from three different IPs and had different MAIL FROMs, they only went to two different destination addresses. The one IP address that sent two messages sent them to different addresses (and in different SMTP sessions, although it was one right after the other).

(In an interesting little detail the most recent message was forged as a bounce message from my own system, although it also had a X-Mailer claiming it had been produced by Outlook Express.)

In contrast to a bunch of copies of the same Chinese spam message that have been sent to message-ids here, all of the destination addresses are at least plausible and two out of the three actually existed at one point.

All of this is what I think of as classical old-fashioned virus behavior that I thought had died out some time ago, partly because so many places had made it hard to get such email through when it was sent directly from end-user machines. After all, any anti-spam system that scored highly based on being on the CBL would have rejected these emails even before running them past virus checking. I guess the old ways are not dead after all, especially if I got five messages within 24 hours of opening my sinkhole server up.

At this point I'll admit I haven't checked our main system's stats recently to see if we're seeing more virus emails there than we used to a year or so ago. If we aren't, I'm not entirely sure what might be causing the difference. While the addresses that these viruses are being spammed to are old addresses, our main system has plenty of equally old addresses (and I believe any number of them get regular spam). Oh well, that's an analysis for another day.

A steady change in the source of blog comment spam attempts

Wandering Thoughts has been in operation for long enough that I've been able to observe a slow shift in the sources of comment spam attempts over the years. Roughly speaking (and relying on a fallible memory), in the beginning much of the comment spam attempts came from what appeared to be open proxies or otherwise compromised machines, to the point where I tried using DNS blocklists like the CBL and SBL as defenses (which didn't work out in the end). Then, at least as I perceived it, the comment spam sources largely shifted to dodgy foreign hosting providers broadly located where you'd expect them to be (Eastern Europe, Russia, and China). And then lately the majority of the still-unblocked sources have shifted to US based hosting providers and datacenters.

At the moment, the largest group of sources seem to emerge from IP address space assigned to 'DataShack LC' and 'WholeSale Internet, Inc'. Where sub-delegation information is readily accessible through whois, the specific IP addresses appear to have been delegated in very small slices to entities that appear to be Chinese based on their names; a typical example is 69.197.128.163, currently assigned to 'Zhou Pizhong' via 69.197.128.160/29. The IP addresses almost never have reverse DNS information available.

For a long time I've been reluctant to explicitly block US hosting providers, for various reasons. I've now decided that that's over for me; large netblocks for these persistent sources are now going in my blocks. Hopefully it will never affect someone using a VPN (or a personal cloud Unix machine) to try to leave a legitimate comment here.

One of several reasons that this depresses me is that it implies that being a source of repeated persistent comment spamming is no longer enough to get people terminated from even US-based hosting (if it ever was). Or at least from second-tier US hosting, since I still don't see much or any comment spam attempts from the large but inexpensive providers like AWS, Linode, and so on.

(I noticed part of this shift to hosting providers a couple of years ago, but back then it was mostly to European hosting providers and many of them were in dodgy areas.)

PS: Mind you, some of this apparent shift in comment spam sources turns out to be a bit illusory. My very first spam comment came from a US hosting provider, as did a lot of sources from a big incident early on. And I haven't kept any sort of records over the years, or even often tried particularly hard to identify the sources and keep notes. The most extensive sort of 'notes' I have are all of the various network areas I've blocked from leaving comments because their volume of comment attempts irritated me, and that's not exactly a scientific process.

TLS versions in connections to my spam-catching sinkhole SMTP server

I've written before about TLS usage on our real inbound mail gateway and the general TLS breakdown on my sinkhole SMTP server. My sinkhole server didn't initially log the TLS version used, but after Heartbleed hit I changed that because it was now interesting information, and here's a preliminary report. Note that, unlike our main mail gateway, this is all spam sending attempts.

The basic version breakdown is that out of 588 connections that negotiated TLS since I added this logging, 102 used SSL v3, 151 used TLS v1.0, only seven used TLS v1.1, and 328 used TLS v1.2. I looked through the seven by hand and there's no particular pattern. Those connections resulted in only 153 actual message submissions, and the breakdown there is 29 SSL v3, 57 TLS v1.0, 2 TLS v1.1, and 65 with TLS v1.2.

I looked through the email received using SSL v3, and almost all of it is basic advanced fee fraud or phishing spam. The first interesting exception is spam from gallopade.nmsrv.com, which is a name that has been trying to spam me for quite a long time. The other interesting exception is genuine email from Twitter's 'please confirm your new account' system and that apparently comes from an account creation initiated by some sort of spammer. Both messages received using TLS v1.1 were also advanced fee fraud emails, although these seem to come from real ISP systems instead of random and probably ancient mail servers around the Internet.

I did a random spot sampling of the messages received with TLS v1.0 and couldn't spot any particular pattern in the mailers involved; connecting to people's SMTP ports turned up Exim, Sendmail, and 'Microsoft ESMTP MAIL Service' aka Microsoft Mail Server before I got bored. Microsoft was clearly the most popular, followed by Exim (although the Exim versions in SMTP greeting banners varied). Interestingly, both GMail and Yahoo used TLS v1.0 at least once.

The prevalence of SSL v3 and some other things in my sinkhole connections goes along with what I've long thought about the machines exploited to send phish and advance fee fraud, which is that many of them are basically neglected. Old, neglected machines are quite likely to be running old software that only supports old versions of TLS.