Wandering Thoughts archives

You should plan for your anti-spam scanner malfunctioning someday

Yesterday I mentioned that the commercial anti-spam and anti-virus system we use ran into a bug where it hung up on some incoming emails. One reaction to this is to point and laugh; silly us for using a commercial anti-spam system, we probably got what we deserved here. I think that this attitude is a mistake.

The reality is that all modern anti-spam and anti-virus systems are going to have bugs. It's basically inherent in the nature of the beast. These systems are trying to do a bunch of relatively sophisticated analysis on relatively complicated binary formats, like ZIP files, PDFs, and various sorts of executables; it would be truly surprising if all of the code involved in doing this was completely bug free, and every so often the bugs are going to have sufficiently bad consequences to cause explosions.

(It doesn't even need to be a bug as such. For example, many regular expression engines have pathological behavior when exposed to a combination of certain inputs and certain regular expressions. This is not a code bug since the RE engine is working as designed, but the consequences are similar.)

What this means is that you probably want to think ahead about what you'll do if your scanner system starts malfunctioning at the level of either hanging or crashing when it processes a particular email message. The first step is to think about what might happen with your overall system and what it would look like to your monitoring. What are danger signs that mean something isn't going right in your mail scanning?

Once you've considered the symptoms, you can think about pre-building some mail system features to let you deal with the problem. Two obvious things to consider are documented ways of completely disabling your mail scanner and forcing specific problem messages to bypass the mail scanner. Having somewhat gone through this exercise myself (more than once by now), I can assure you that developing mailer configuration changes on the fly as your mail system is locking up is what they call 'not entirely fun'. It's much better to have this sort of stuff ready to go in advance even if you never turn out to need it.

(Building stuff on the fly to solve your urgent problem can be exciting even as it's nerve-wracking, but heroism is not the right answer.)

At this point you may also want to think about policy issues. If the mail scanner is breaking, do you have permission to get much more aggressive with things like IP blocks in order to prevent dangerous messages from getting in, or is broadly accepting email important enough to your organization to live with the added risks of less or no mail scanning? There's no single right answer here and maybe the final decisions will only be made on the spot, but you and your organization can at least start to consider this now.

You should probably track what types of files your users get in email

Most of the time our commercial anti-spam system works fine and we don't have to think about it or maintain it (which is one of the great attractions of using a commercial system for this). Today was not one of those times. This morning, we discovered that some incoming email messages we were receiving make its filtering processes hang using 100% CPU; after a while, this caused all inbound email to stop. More specifically, the dangerous incoming messages appeared to be a burst of viruses or malware in zipped .EXEs.

This is clearly a bug and hopefully it will get fixed, but in the mean time we needed to do something about it. Things like, say, blocking all ZIP files, or all ZIP files with .EXEs in them. As we were talking about this, we realized something important: we had no idea how many ZIP files our users normally get, especially how many (probably) legitimate ones. If we temporarily stopped accepting all ZIP file attachments, how many people would we be affecting? No one, or a lot? Nor did we know what sort of file types are common or uncommon in the ZIP files that our users get (legitimate or otherwise), or what sort of file types users get other than ZIP files. Are people getting mailed .EXEs or the like directly? Are they getting mailed anything other than ZIP files as attachments?

(Well, the answer to that one will be 'yes', as a certain amount of HTML email comes with attached images. But you get the idea.)

Knowing this sort of information is important for the same reason as knowing what TLS ciphers your users are using. Someday you may be in our situation and really want to know if it's safe to temporarily (or permanently) block something, or whether it'll badly affect users. And if something potentially dangerous has low levels of legitimate usage, well, you have a stronger case for preemptively doing something about it. All of this requires knowing what your existing traffic is, rather than having to guess or assume, and for that you need to gather the data.

Getting this sort of data for email does have complications, of course. One of them is that you'd really like to be able to distinguish between legitimate email and known spam in tracking this sort of stuff, because blocking known spam is a lot different than blocking legitimate email. This may require logging things in a way that either directly ties them to spam level information and so on or at least lets you cross-correlate later between different logs. This can affect where you want to do the logging; for example, you might want to do logging downstream of your spam detection system instead of upstream of it.

(This is particularly relevant for us because obviously we now need to do our file type blocking and interception upstream of said anti-spam system. I had been dreaming of ways to make it log information about what it saw going by even if it didn't block things, but now maybe not; it'd be relatively hard to correlate its logs again our anti-spam logs.)

Today's odd spammer behavior for sender addresses

It's not news that spammers like to forge your own addresses into the MAIL FROMs of the spam that they're trying to send you; I've seen this here for some time. On the machine where I have my sinkhole server running, this clearly comes and goes. Some of the time almost all the senders will be trying a legitimate MAIL FROM (often what they seem to be trying to mail to), and other times I won't see any in the logs for weeks. But recently there's been a new and odd behavior.

Right now, a surprising number of sending attempts are using a MAIL FROM that is (or was) a real address, but with the first letter removed. If 'joey@domain' was once a real address, they are trying a MAIL FROM of 'oey@domain'. They're not just picking on a single address that is mutilated this way, as I see the pattern with a number of addresses.

(Some of the time they'll add some letters after the login name too, eg 'joey@domain' will turn into 'oeyn@domain'.)

So far I have no idea what specific spam campaign this is for because all of the senders have been in the Spamhaus XBL (this currently gets my sinkhole server to reject them as boring spam that I already have enough samples of).

What really puzzles me is what the spammers who programmed this are thinking. It's probably quite likely that systems will reject bad local addresses in MAIL FROMs for incoming email, which means that starting with addresses you think are good and then mutating them is a great way to get a lot of your spam sending attempts rejected immediately. Yet spammers are setting up their systems to deliberately mutate addresses and then use them as the sender address, and presumably this both works and is worthwhile for some reason.

(Perhaps they're trying to bash their way through address obfuscation, even when the address isn't obfuscated.)

(I suspect that this is a single spammer that has latched on to my now spamtrap addresses, instead of a general thing. Our general inbound mail gateway gets too much volume for me to pick through the 'no such local user' MAIL FROM rejections with any confidence that I'd spot such a pattern.)

SPF is not a security feature, as it solves the wrong problem

SPF is one of my hot button issues, or rather how all too often influential people seem to think that SPF is a good idea. A lot of the time these people seem to think that a hard-fail SPF policy is a security feature, something that will prevent forgery of email as being from their company or organization. These people are wrong, at least in any practical sense.

The problem with SPF as a security feature is that it protects the wrong thing. To the extent that it does anything, SPF protects the (SMTP) envelope sender, ie the MAIL FROM domain, and the envelope sender is effectively invisible to people reading their email. I am an email expert and even I do not configure my mail client to display the envelope sender; like everyone else, I see the From: header. Ordinary people generally don't even know that a separate envelope sender address even exists.

What this means is that an attacker who wants to forge email from your domain is not at all deterred by your hard-fail SPF policy. They just put something else in the envelope sender, put your domain in the From:, and mail away. It's extremely unlikely that anyone will notice anything or that any automated systems will lower the reputation score of these forged email messages (at least for that reason). And I'm being extremely generous here, since I'm assuming that people even see or look at the domain of the From: address, as opposed to simply seeing some user-friendly version of it that may be based on, eg, the name in the From: instead of the domain.

(For example, GMail will show you the domain of the From: but it seems to de-emphasize it, using smaller type in a lighter shade compared to the person's display name. If people aren't already suspicious, how likely are they to notice a mismatch in such a thing?)

If you want a security feature that tries to block people forging your domain in a meaningful sense, you want DMARC. DMARC specifically exists to protect the From: domain and in the process the integrity of your legitimate email, so that it can't be either forged or altered. SPF has nothing to do with this. Of course even preventing forged From: domains is not a great protection, but at least DMARC does something useful with only moderate collateral damage, unlike hard-fail SPF.

(SPF does not really solve any problem, especially these days. The one problem it might solve it doesn't because lots of MTAs sensibly ignore it. See the sidebar here and of course SPF also has major downsides.)