Today, we discovered that we were being exploited to send out a batch of spam. I decided to write up some information about the incident, partly because I don't think I've seen this done very much and who knows, it might be useful to other people. Much of this information is (very) preliminary, since this just happened and it's the weekend so we haven't done any deep investigation yet.

(As peculiar as it may seem for sysadmins, around here we try to take the weekends off.)

So far it looks like only a single local account was used to send the spam. There were two recent large-scale targeted phish spam runs against our users, so it's a decent guess that the user's account was compromised through one of them (we won't know for sure until we've talked to the user, which is probably going to be Monday or later).

The spam incident itself lasted about two and a half hours, ending when we noticed it and started turning things off. During those two and a half hours the spammers generated and sent (I think) 2,339 spam messages, which is a pretty impressive rate (over 14 a minute). The spam seems to have been two versions of your standard advance fee fraud spam (of the 'you have won a prize' variant). All of the spam messages went out with forged origin addresses; one version used info@exxon.fr and the other version used web@live.com.

All of the spamming was done through our webmail system, and there's no current evidence that the compromised user's account was accessed in any other way. Here's where it gets interesting. We have two webmail systems, a newer one using Roundcube and an older one using SquirrelMail that we're migrating away from. Although the Roundcube one is the default webmail environment and the spammers poked through it, they chose to send all of their mail through SquirrelMail instead. I'm going to skip all sorts of speculation about why, at least for now.

The actual spamming run itself was done using multiple ProXad IP addresses, in fact multiple IP addresses connected to webmail at once; this suggests either automation or that the lead spammer had a bunch of 'mules' doing the grunt-work of entering and sending messages (which would certainly help to enter 14 messages a minute). The lead spammer is likely in Nigeria; before the spam run from ProXad IPs started there was a connection to this user's webmail account from a Nigerian IP address (looking around both webmail systems before apparently settling on SquirrelMail).

Unfortunately I suspect that we can look forward to more incidents of this nature, since it seems really optimistic to assume that only one user's account was compromised in this phishing attack. We may wind up with an environment where we filter outgoing mail after all.