Some brief information about a local spam incident

August 26, 2012

Today, we discovered that we were being exploited to send out a batch of spam. I decided to write up some information about the incident, partly because I don't think I've seen this done very much and who knows, it might be useful to other people. Much of this information is (very) preliminary, since this just happened and it's the weekend so we haven't done any deep investigation yet.

(As peculiar as it may seem for sysadmins, around here we try to take the weekends off.)

So far it looks like only a single local account was used to send the spam. There were two recent large-scale targeted phish spam runs against our users, so it's a decent guess that the user's account was compromised through one of them (we won't know for sure until we've talked to the user, which is probably going to be Monday or later).

The spam incident itself lasted about two and a half hours, ending when we noticed it and started turning things off. During those two and a half hours the spammers generated and sent (I think) 2,339 spam messages, which is a pretty impressive rate (over 14 a minute). The spam seems to have been two versions of your standard advance fee fraud spam (of the 'you have won a prize' variant). All of the spam messages went out with forged origin addresses; one version used and the other version used

All of the spamming was done through our webmail system, and there's no current evidence that the compromised user's account was accessed in any other way. Here's where it gets interesting. We have two webmail systems, a newer one using Roundcube and an older one using SquirrelMail that we're migrating away from. Although the Roundcube one is the default webmail environment and the spammers poked through it, they chose to send all of their mail through SquirrelMail instead. I'm going to skip all sorts of speculation about why, at least for now.

The actual spamming run itself was done using multiple ProXad IP addresses, in fact multiple IP addresses connected to webmail at once; this suggests either automation or that the lead spammer had a bunch of 'mules' doing the grunt-work of entering and sending messages (which would certainly help to enter 14 messages a minute). The lead spammer is likely in Nigeria; before the spam run from ProXad IPs started there was a connection to this user's webmail account from a Nigerian IP address (looking around both webmail systems before apparently settling on SquirrelMail).

Unfortunately I suspect that we can look forward to more incidents of this nature, since it seems really optimistic to assume that only one user's account was compromised in this phishing attack. We may wind up with an environment where we filter outgoing mail after all.

Comments on this page:

From at 2012-08-27 04:16:41:

I've seen this several times as well. It always follows the exact same pattern, first a targeted phishing attack to obtain usernames and passwords and then they use the available webmail interface to send spam.

We block the phishing mails as soon as possible, but by then its already to late. Thousands will have gotten through. We also block all the responses to that message to reduce the number of compromised accounts. Of course the spammers are smart enough to launch these attacks during the weekend when response times are typically slower.

Outbound spam scanning has been active here for years. In fact, we usually notice these events just because the queues fill up. There is no easy way to fix this.. other than rate-limiting requests to webmail and any other system that can be used in a similar manner.

From at 2012-09-02 01:01:03:

When I worked at a University, we saw the same issue. The phishing attempts (as well as the resultant spam messages) all started on a Friday or Saturday afternoon/evening with the idea that we wouldn't be around to detect them and sufficiently block them.

After a few incidents which caused our sender reputation to drop (and subsequently cause legitimate outgoing messages to be denied by various mail servers that bothered to check), we implemented a system which would add a header to any outgoing mail from any user that exceeded a particular quota per day. We then used our existing outgoing spam system to look for that header and quarantine any messages exceeding the quota.

We also setup monitoring to alert when we started quarantining messages based on this header, so that we could quickly determine if the messages were legitimate or indeed spam (allowing us to update filters, reset passwords on accounts, notify users, etc). This also allowed us to keep the filter list on outgoing mail small in comparison to our incoming mail filters, since we had a lot of outgoing legitimate messages that tended to look like spam to our normal set of incoming filters (fundraising, survey, and departmental update emails in particular).

Written on 26 August 2012.
« Some odd behavior from blog comment spammers
You should log all successful user authentication »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Aug 26 03:23:13 2012
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.