Wandering Thoughts archives

Weekly spam summary on March 24th, 2007

This week, we:

• got 12,733 messages from 249 different IP addresses.
• handled 21,567 sessions from 1,259 different IP addresses.
• received 197,829 connections from at least 58,846 different IP addresses.
• hit a highwater of 8 connections being checked at once.

This is up from last week, although the messages received count remains down from the usual levels.

Day	Connections	different IPs
Sunday	29,942	+11,252
Monday	30,350	+9,115
Tuesday	29,884	+8,506
Wednesday	28,581	+8,341
Thursday	23,071	+7,064
Friday	30,424	+8,093
Saturday	25,577	+6,475

This has an interesting general decline in the number of new different IP addresses talking to us over the week (and the general Thursday dip also makes me wonder).

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
72.249.13.82          19714   1084K
68.230.240.0/24       15468    751K
213.4.149.12          13923    724K
213.29.7.0/24         11192    672K
205.152.59.0/24       10640    482K
69.25.186.89           4493    216K
81.115.40.8            3979    212K
24.97.42.82            3645    170K
72.32.54.146           2383    143K
211.75.135.252         2245    135K

This is down from last week, partly because at least some of the active webmail subnets seem to have quieted down a bit.

• 72.249.13.82 and 72.32.54.146 kept trying to send stuff with origin addresses that had tripped our spamtraps.
• 213.4.149.12 reappears from last week.
• 69.25.186.89 is in acceleratebiz.com IP address space, and we don't talk to that any more. Considering its current hostname is 'mail.thefreebiediscount.com', I can't imagine that we're missing much.
• 81.115.40.8 is a telecomitalia.it IP address and returns from earlier this month.
• 24.97.42.82 kept trying with a bad HELO name.
• 211.75.135.252 is a Taiwanese IP address with no reverse DNS.

Connection time rejection stats:

  62832 total
  38554 dynamic IP
  17429 bad or no reverse DNS
   5222 class bl-cbl
    262 acceleratebiz.com
    185 class bl-sbl
    160 class bl-pbl
    154 class bl-sdul
    127 dartmail.net
    101 class bl-dsbl
     94 cuttingedgemedia.com
     72 class bl-njabl

(Note that I don't always put specific domain blocks in this list, even if they show up in the overall numbers.)

The highest SBL source this week is SBL52715 (a spam source and landing pages /27, listed only today) at 108 rejections. Next is SBL50181 (good old microcamp.com.br's compromised web server, listed since January 18th) at 37 rejections.

Nine of the top 30 most rejected IP addresses were rejected 100 times or more this week; the leaders are 85.98.39.192 (455 rejections, bad reverse DNS), 81.208.36.80 (247 rejections, generic fastwebnet.it), 200.193.90.196 (221 rejections, bad reverse DNS), and 70.107.170.22 (217 rejections, verizon dynamic IP). It's striking that only two out of the nine are not in zen.spamhaus.org.

Fourteen of the top 30 are currently in the CBL, twelve are currently listed in bl.spamcop.net, fourteen are currently in the PBL, and a grand total of 20 are in zen.spamhaus.org.

This week Hotmail had:

• no messages accepted.
• no messages rejected because they came from non-Hotmail email addresses.
• 32 messages sent to our spamtraps.
• no messages refused because their sender addresses had already hit our spamtraps.
• 5 messages refused due to their origin IP address (two in the CBL, two from Nigeria, and one in SBL49971.)

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	561	81	556	79
Bad bounces	2	2	13	7

Now those are the sort of numbers on bad bounces that I like to see. As usual, bad HELOs have no sources that particularly stand out; the highest is 64.163.170.34 (63 rejections).

Bad bounces were sent to two different bad usernames this week. Both went to plausible usernames that have never existed here (to the best of my memory), and this week they both came from machines in the USA.

How comment spammers behave

One of the things that watching your logs while trying out various comment spam precautions is good for is seeing how comment spammers seem to behave, or at least how the comment spammers that drop by WanderingThoughts behave. (Your mileage may vary, since there are a lot of comment spammers out there and they can't all be using the same tools.)

As before, I'm only really interested in defeating the automated comment spammers; a dedicated person is always going to be able to leave comments here. (And I'm not interested in making it so that people writing comments can't include links.)

So, my observations on comment spammers to date:

• they will hit any POST form with a submit button that they can see. They don't seem to spam the search box (which is a GET form without an explicit submit button), but they do regularly try to submit comment spam through DWiki's login form.

  (The most amusing login form spammer is the one that believes in being honest; they start all of their spam attempts with 'sorry, but i need money...'.)

• however, they almost never go past the first form submission. The single greatest reduction in successful comment spam that I ever managed was changing my comment form so that you had to preview before actually posting your comment; almost every spammer previewed and then just went away.

• some but not all of them fill in any form field that they spot; my comment form's honeypot field gets a regular stream of programs that trip over it, but there are about as many spammers who don't.

• the basic User-Agent checking I do is surprisingly effective. It is also a very cheap check to make, since you can even do it in Apache itself.

• a fair number of them harvest your comment form from one IP and then submit from another (or a pool of others). This is really easy to see in the full web logs, and so my 'must submit from the same /24' precaution trips up a reasonable number of would-be comment spammers.

  However, the really interesting thing is that a number of comment spammers modify this hidden field. All of the spammers that modify it seem smart enough to try putting in IP addresses, but they make them up randomly instead of using the IP address they're POSTing the form from, and they don't notice that the field is not formatted as a straight IP address. (And sometimes they stick some newlines on the end.)

  They may be doing this partly because I called the field 'previp'. (My current format for it is the IP address less the last octet, so the real version looks like 'A.B.C.', with no newline at the end.)

  Looking at some numbers, it appears that most comment spammers that don't trip up on the honeypot field make up random IP addresses to put in this field instead of leaving it alone.

• comment spammers almost always use comment spam using all four of the popular syntaxes for making links at once. These seem to be:
  1. a bare URL: http://...
  2. a full HTML link: <a href="http://...">..</a>
     (I have seen one spammer that turned the initial < into &lt;.)
  3. [url]http://...[/url]
  4. [url=http://...]...[/url]

  (I'm not sure what uses the last two forms, but they turn up a lot.)

  The links don't necessarily all go to the same website, but the presence of all four forms in the same comment is a pretty good danger sign.

  (As the result of a recent aggressive (and temporarily successful) spam run, WanderingThoughts currently rejects comments that contain any of the last three forms of links, since they don't work here anyways.)

• while typical comment spam attempts to include more links than normal, it's not a lot more than normal; for example, the recent aggressive comment spam only had four links per comment (one in each link format).

I also have some negative results. First, it's not worth checking for correct Referer values; almost every comment spammer that made it past my basic User-Agent checks sent the right value.

Also, very soon after I changed my comment form to only have a preview option at the start I saw a significant jump in comment spam attempts. From this I formed the hypothesis that comment spammers are unduly attracted to forms with only one submit button; however, various experiments I've tried since then suggest that this isn't the case.

(I changed things so the first 'add comment' page had two form submission buttons and the backend DWiki code just made them do the same thing. But I didn't see any reduction in comment spam attempts, even across various variants of how the buttons were named and so on.)

Weekly spam summary on March 17th, 2007

This week, we:

• got 11,732 messages from 232 different IP addresses.
• handled 18,216 sessions from 1,165 different IP addresses.
• received 189,951 connections from at least 55,941 different IP addresses.
• hit a highwater of 7 connections being checked at once.

This is all down from last week, and I have no explanation for why the messages received count is down so much; it is normally quite stable.

Day	Connections	different IPs
Sunday	15,731	+6,525
Monday	30,676	+9,666
Tuesday	28,663	+8,088
Wednesday	29,394	+8,296
Thursday	32,932	+8,916
Friday	29,720	+8,318
Saturday	22,835	+6,132

The Sunday count is unnaturally low because we managed to accidentally drop the machine off the network for about eight hours on Sunday (we had a mis-set default route in the configuration files, so when the regular Sunday morning reboot happened the machine dropped off the Internet until we figured out what was going on).

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
206.223.168.238       34311   1882K
68.230.240.0/24       23616   1147K
213.4.149.12          17757    923K
205.152.59.0/24       17549    796K
213.29.7.0/24         12251    735K
213.41.128.40          7368    375K
70.167.3.24            6324    379K
69.15.68.98            6321    296K
211.63.211.245         5964    286K
217.14.208.79          5586    284K

This is significantly up from last week, partly (but not entirely) because of 68.230.240.0/24, which is Cox's outgoing SMTP pool. Cox is yet another US ISP that we don't talk to any more because they got into full bore webmail and thus full bore advance fee fraud spamming, and this week I blocked their /24 early on.

• 206.223.168.238 and 213.4.149.12 return from last week and previous appearances.
• 213.41.128.40, 211.63.211.245, and 217.14.208.79 are all on the DSBL.
• 70.167.3.24 kept trying to send stuff with an origin address that had already tripped our spamtraps.
• 69.15.68.98 kept trying with a bad HELO name; we've seen it before, back in early February.

To follow up something from last week: 64.208.191.0/24 did not hit us at all this week, and thus I am dropping them off my mental radar.

Connection time rejection stats:

  67425 total
  41908 dynamic IP
  17325 bad or no reverse DNS
   6573 class bl-cbl
    299 class bl-dsbl
    245 acceleratebiz.com
    242 class bl-sdul
    159 class bl-pbl
     93 class bl-njabl
     85 cuttingedgemedia.com
     49 class bl-sbl

The highest SBL source this week is SBL43107 (18 hits), the 'Gestour Portal spam source' listing that we've seen before. After that is SBL49248 (9 hits), an advance fee fraud spam source listed 18 December 2006.

Three of the top 30 most rejected IP addresses were rejected 100 times or more this week; the leader is 66.191.255.223 (112 times), a charter.com dynamic IP address of some sort. Twelve of the top 30 are currently in the CBL, 13 are currently in bl.spamcop.net, eight are in the PBL, and a grand total of 16 are in zen.spamhaus.org (which needs a short, punchy name).

This week Hotmail managed:

• 2 messages accepted; I suspect both were spam.
• no messages rejected because they came from non-Hotmail email addresses.
• 36 messages sent to our spamtraps.
• 1 message refused because its sender address had already hit our spamtraps.
• 5 messages refused due to their origin IP address (one in SBL33955 (which dates from 2005), one in SBL47589), one in the CBL, one from the Cote d'Ivoire, and one from Ghana).

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	555	78	1041	96
Bad bounces	13	7	4	4

The numbers on bad bounces have gotten a bit worse, but only a bit. Bad HELOs had no really big sources; the biggest three were 65.120.172.122 (71 tries), 72.54.106.163 (63 tries), and 74.62.160.114 (50 tries).

One machine contributed more than half of the bad bounces this week; 72.37.163.14 tried to send seven bounces to a single bad username. Bad bounces were sent to 6 different usernames this week, all of them ex-users. One ex-user got eight bounces; all the others got one each.

Weekly spam summary on March 10th, 2007

This week, we:

• got 14,862 messages from 263 different IP addresses.
• handled 21,019 sessions from 1,246 different IP addresses.
• received 197,155 connections from at least 66,752 different IP addresses.
• hit a highwater of 11 connections being checked at once.

Volume is definitely down from last week, although the session volume is up slightly. The per day numbers have some significant fluctuations:

Day	Connections	different IPs
Sunday	32,593	+13,521
Monday	37,161	+12,938
Tuesday	27,967	+8,530
Wednesday	21,632	+7,902
Thursday	29,415	+9,045
Friday	28,617	+8,658
Saturday	19,770	+6,158

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
206.223.168.238       39471   2165K
213.29.7.0/24         24821   1488K
205.152.59.0/24       14936    677K
213.4.149.12          11163    581K
66.16.116.241          5392    259K
71.8.237.114           5374    273K
81.215.229.141         3248    156K
64.83.75.188           3207    154K
64.208.191.0/24        3121    187K
193.202.89.232         2898    155K

This is up from last week, although it's unevenly distributed; the low end is about the same, but the high end is much more active.

• 206.223.168.238 and 213.4.149.12 return from last week.
• 66.16.116.241 kept trying with a bad HELO.
• 71.8.237.114 is a charter.com something or other.
• 81.215.229.141 has inconsistent reverse DNS.
• 64.83.75.188 is a place we no longer talk to because it sent us phish spam.
• 193.202.89.232 kept trying with an origin address that had already tripped our spamtraps.

64.208.191.0/24 deserves special mention: various hosts in there slammed us as part of an aggressive spam run, and then once they had tripped our spamtraps they demonstrated that they were partially ignoring SMTP responses. This is a quick recipee for getting your own set of kernel packet filtering rules; if they come back this week, I'll probably make the block permanent.

Connection time rejection stats:

  62956 total
  37668 dynamic IP
  17559 bad or no reverse DNS
   5173 class bl-cbl
   1049 class bl-sbl
    353 acceleratebiz.com
    198 class bl-dsbl
    132 class bl-pbl
    121 cuttingedgemedia.com
    112 class bl-sdul
     78 class bl-njabl
     27 verticalresponse.com

Overall volume is slightly down from last week. The SBL breakdown is as uninteresting as last week; 962 hits from SBL50892 (colocentral.com, who apparently feel spammer hosting is fine with them), then the next highest is 18 hits from SBL43107 (listed February 16th as 'Gestour Portal spam source').

Four of the top 30 most rejected IP addresses were rejected 100 times or more: 81.51.111.171 (2,190 times, a wanadoo.fr dynamic IP address), 200.88.30.51 (114 times, no reverse DNS), 24.158.104.204 (106 times, a charter.com cablemodem or something), and 71.101.60.68 (106 times, a verizon.net DSL something or other). Fourteen of the top 30 are currently in the CBL, 11 are currently listed by bl.spamcop.net, 11 are in the Spamhaus PBL, and a grand total of 17 are in zen.spamhaus.org.

This week Hotmail managed:

• 3 messages accepted.
• no messages rejected because they came from non-Hotmail email addresses.
• 26 messages sent to our spamtraps.
• no messages refused because their sender addresses had already hit our spamtraps.
• 1 message refused due to its origin IP address being in the CBL.

This is a lot better than their numbers last week.

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	1041	96	953	95
Bad bounces	4	4	17	16

Now that's the sort of numbers on bad bounces that I like to see. There were no really big sources of bad HELOs this week; the highest were 64.3.170.46 (113 times), 69.15.31.193 (82 times), 64.122.66.34 (76 times), and 64.171.104.2 (75 times).

Bad bounces came from four different places to four different usernames; three of the bad usernames are ex-users, and one is a reasonably plausible username.

Why hashcash schemes for email will never be adopted

Hashcash schemes are an attempt to slow down the rate that spammers can send email by making them burn CPU time on some non-trivial calculation. This sounds superficially attractive (and has ever since 1997), but will never be adopted; in fact, the idea is dead on arrival.

To see why, we only need to observe that hashcash has a disproportionate effect on the heaviest senders of email; in fact, this is exactly what it is designed to do. Now think of what this does to the infrastructure needs of the sources of the largest amounts of legitimate email on the Internet today, namely the large ISPs like AOL, Hotmail, Yahoo, Google, Comcast, Bellsouth, and so on.

These ISPs are not going to start generating hashcash on outgoing email, no matter what; they just can't afford to. Now, practically by definition these are large sources of email your users want and you can't afford to block, which means that you can't use the absence of hashcash to block email. (Or to score it as spam, which is effectively the same thing.)

Fundamentally, hashcash schemes only work for places that don't send much email. But those are (on average) the unimportant sources of email, not the important ones; the important ones are the high volume senders.

Schemes that only work for unimportant sources of email don't get adopted; there's no point.

(The alert reader will observe that much of this argument also applies to the perennial anti-spam proposal of charging places real money to send email.)

(Hashcash wouldn't actually work even if adopted for reasons adequately covered in the paper 'Proof-of-Work' Proves Not to Work [PDF] that the Wikipedia Hashcash page has a link to.)

Weekly spam summary on March 3rd, 2007

This machine had a planned twelve hour power outage today, so many of these statistics are really only for six days. Having said that, this week we:

• got 16,376 messages from 272 different IP addresses.
• handled 20,396 sessions from 1,270 different IP addresses.
• received at least 212,857 connections from at least 63943 different IP addresses.
• hit a highwater of 5 connections being checked at once.

This is down from last week, but not hugely so; we might have been in the same ballpark if not for the downtime.

Day	Connections	different IPs
Sunday	35,230	+12,485
Monday	32,638	+10,273
Tuesday	38,623	+11,238
Wednesday	34,186	+10,274
Thursday	36,476	+10,272
Friday	31,556	+9,401

This is reasonably similar to last week's, although smoother.

Kernel level packet filtering top ten (up to 02:26 am on March 3rd):

Host/Mask           Packets   Bytes
205.152.59.0/24       10914    495K
206.223.168.238        9216    505K
213.4.149.12           6660    346K
69.25.186.66           5673    272K
81.115.40.8            5360    286K
213.29.7.0/24          4317    259K
68.22.111.226          4051    189K
65.14.221.82           3569    171K
204.202.15.102         3019    149K
211.94.0.0/15          2919    175K

This is down significantly from last week, and it seems unlikely that one more day would have made a major difference.

• 205.152.59.0/24 is Bellsouth, still hammering on us with advance fee fraud spammers through their webmail system. (Well, probably. Since we're not accepting their packets I can't be sure.)
• 206.223.168.238 and 204.202.15.102 return from last week.
• 213.4.149.12 returns from recently and is resuming its usual presence in the listing.
• 69.25.186.66 is mail.mydiscountoffer.com, and was blocked for being in AccelerateBiz network space; after too many spammers, we no longer accept connections from their IP ranges.
• 81.115.40.8 is a telecomitalia.it IP address, last seen in January.
• 68.22.111.226 and 65.14.221.82 kept trying with bad HELOs.

Connection time rejection stats:

  66671 total
  40920 dynamic IP
  16846 bad or no reverse DNS
   5790 class bl-cbl
   1512 class bl-sbl
    462 acceleratebiz.com
    225 class bl-pbl
    109 class bl-njabl
    104 class bl-dsbl
     79 cuttingedgemedia.com
     64 class bl-sdul

This is pretty close to last week, and might even have been over it if not for the 12 hour downtime. I'd do a breakdown of the SBL rejections, but there's no real point; 1440 of them come from SBL50892, which is a colocentral.com spammer hosting escalation listing from Feburary 6th, and the next highest one is 12 rejections. (The colocentral.com rejections were spread over 248 different IP addresses, with none of them having more than 9 rejections. The hostnames suggest that we didn't miss anything.)

Three of the top 30 most rejected IP addresses were rejected 100 times or more this week: 69.25.186.66 (181 times), 67.102.251.238 (176 times, a Covad something or other), and 210.176.52.139 (149 times, no reverse DNS). Twelve of the top 30 are currently in the CBL, ten are currently in bl.spamcop.net, eight are in the PBL, and a grand total of 15 of the 30 are in zen.spamhaus.org.

This week Hotmail did:

• 4 messages accepted, at least two of them legitimate and one almost certainly spam.
• no messages rejected because they came from non-Hotmail email addresses.
• 41 messages sent to our spamtraps.
• 30 messages refused because their sender addresses had already hit our spamtraps.
• 8 messages refused due to their origin IP address (4 in the CBL, 3 from the Cote d'Ivoire, and one in SBL45516).

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	953	95	877	101
Bad bounces	17	16	16	12

There was no particularly flagrant source of bad HELOs this week, just the usual crowd with middle double digit rejections before we dumped them in the kernel filters. Bad bounces once again came from all over, although possibly with more North American sources than anywhere else.

Bad bounces were sent to 15 different usernames this week, once again mostly to real ex-users and plausible usernames (and one valid ex-user with some numbers glued on the front). The most popular target, with three bounces, was an ex-user.