Wandering Thoughts archives

Some words of wisdom on email certification programs

Vernon Schyrver in news.admin.net-abuse.email:

The trouble with all "Good Guys" lists is in who pays. Maintaining a comprehensive list that can be widely used is expensive. [...]

Sooner instead of later, those who have financial reasons for seeing themselves certified as good assume the financial burdens and so are given control. Organizations whose mail is never blocked can't find a reason to pay third parties for testamonials. They tell their correspodents "If you want my mail, orders or whatever, whitelist me or otherwise arrange to hear from me, but if not, that's ok too." That leaves organizations that for various reasons don't inspire enough trust on their own to pay for and so run the "Good Guys" list.

(From Message-ID <eittga$26l6$1@calcite.rhyolite.com>.)

Weekly spam summary on November 25th, 2006

This week, we:

• got 13,435 messages from 269 different IP addresses.
• handled 21,900 sessions from 1,620 different IP addresses.
• received 197,361 connections from at least 50,069 different IP addresses.
• hit a highwater of 8 connections being checked at once.

This is down from last week, although the connection numbers are still up a bit compared to two weeks ago.

Day	Connections	different IPs
Sunday	28,053	+6,271
Monday	35,445	+9,698
Tuesday	34,715	+8,208
Wednesday	28,591	+6,410
Thursday	25,680	+6,815
Friday	25,068	+6,826
Saturday	19,809	+5,841

This shows no signs of any lingering surge from last week's huge wave; the fluctuations are pretty typical.

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
213.29.7.0/24         21761   1306K
207.235.5.169         13648    819K
213.4.149.12           9110    474K
208.99.198.92          7501    450K
208.99.198.71          6627    398K
208.99.198.93          6601    396K
64.65.197.93           6484    303K
208.99.198.94          6235    374K
64.166.14.222          5667    272K
208.99.198.66          5393    324K

This is a rather different week:

• 213.29.7.0/24 is where centrum.cz's outgoing mail servers live. We've received too much advance fee fraud spam from them to be interested in talking to them, and I've moved up to blocking their /24 rather than play whack-a-mole with individual active IPs there.
• 207.235.5.169 is 'potter.aper.net' and kept trying to send us stuff with an origin address that had already tripped our spamtraps.
• 213.4.149.12 is terra.es, returning from last week and many, many times before.
• all of 208.99.198.64/27 is that rarity, an active SBL listed spammer; they are SBL48200, aka 'totallyfreeld.net', aka 'The Client Store'. They get upstream connectivity and their IP address space from 'Swift Ventures Inc', aka swiftco.net, which currently has a number of SBL listings, one for a ROKSO spammer dating back to June 3rd.
• 64.65.197.93 kept trying with a bad HELO name.
• 64.166.14.222 also reappears from last week; it is a PacBell DSL line.

If all of SBL48200 was counted together, it would be the leader at 58061 packets or so. (We blocked them one by one this week, so that's how they get counted up here.)

This is clearly up from last week, with multiple highly active would-be senders, some of them repeats (especially the terra.es machine, which just keeps trying like a certain commercialized rabbit).

Connection time rejection stats:

  42352 total
  24394 dynamic IP
  13850 bad or no reverse DNS
   2504 class bl-cbl
    332 class bl-dsbl
    228 class bl-sdul
    156 class bl-sbl
     99 class bl-njabl
     62 class bl-spews
     37 class bl-ordb
     22 cuttingedgemedia.com

Evidently we caught SBL48200 early enough that they didn't make a big dent in the SBL numbers.

Four of the top 30 most rejected IP addresses were rejected 100 times or more; the leader is our friend 64.166.14.222 (at 452 times, which right away means that it has some problems), followed by an internal IP address, 211.52.83.131 (177 times) and 70.68.66.159 (110 times). 19 of the top 30 are currently in the CBL, and 3 are currently in bl.spamcop.net.

This week's Hotmail numbers:

• 4 messages accepted; at least two of them were spam.
• no messages rejected because they came from non-Hotmail email addresses.
• 31 messages sent to our spamtraps.
• 2 messages refused because their sender addresses had already hit our spamtraps.
• 3 messages refused due to their origin IP address (two from Nigeria, one for being in the SBL (and also located in Nigeria, apparently)).

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	2148	161	2008	175
Bad bounces	412	319	470	374

Despite the change in accounting from last week, the bad bounce stats haven't particularly jumped; I guess that most places already weren't trying to send us multiple bad bounces in one SMTP transaction.

This week's champion bad bounce target was 'kiseleeva_margarita', at 24 tries. My friend 3E4B made a return, and first_last stuff seems to be up from last week. The top two sources of bad bounces are it.valmi.com.ua (217.25.199.249) and verify-sender.lucky.net (193.193.193.135), along with a pile of similarly located servers, so it seems that we're a popular forgery target with spammers targeting Eastern Europe.

Weekly spam summary on November 18th, 2006

This week, we:

• got 15,266 messages from 269 different IP addresses.
• handled 22,009 sessions from 1,693 different IP addresses.
• received 897,934 connections from at least 48,417 different IP addresses.
• hit a highwater of 10 connections being checked at once.

Just like last time, that is not a typo (and nothing on the system was affected enough to make me notice it before now, when my scripts started spitting out information for this). Apart from the huge connection volume, things are pretty much the same as last week; ironically, the highwater is lower.

Day	Connections	different IPs
Sunday	27,857	+7,529
Monday	25,752	+6,652
Tuesday	32,064	+7,395
Wednesday	32,050	+7,526
Thursday	173,814	+7,585
Friday	312,437	+6,042
Saturday	293,960	+5,688

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
64.166.14.222         12859    617K
213.4.149.66           7756    403K
212.216.176.0/24       6340    312K
213.4.149.12           6325    329K
61.128.0.0/10          5730    314K
216.102.133.220        5532    259K
64.216.74.90           4732    227K
67.153.140.35          4396    224K
203.113.158.222        4261    205K
210.50.231.134         3947    181K

Overall things are quieter than last week.

• 64.166.14.222 and 213.4.149.12 return from last week.
• 213.4.149.66 and 210.50.231.134 got blocked for repeated bad HELOs.
• 216.102.133.220 is listed in dul.dnsbl.sorbs.net, although its hostname and WHOIS information doesn't look like a dynamic IP address. My best theory is that SORBS just listed the large SBC Internet block that it's in.
• 64.216.74.90 was blocked for repeatedly trying to send us phish spam.
• 67.153.140.35 is an algx.net dynamic IP address.
• 203.113.158.222 is a Vietnamese IP address with no reverse DNS.

Connection time rejection stats:

  46168 total
  26202 dynamic IP
  15926 bad or no reverse DNS
   2374 class bl-cbl
    317 class bl-sdul
    285 class bl-dsbl
    206 class bl-njabl
     89 class bl-spews
     76 cuttingedgemedia.com
     40 class bl-sbl
     38 class bl-ordb

Good old Cutting Edge Media; apparently they never give up. But then, this is completely typical of spammers. That these numbers are not hugely larger than last week suggests that most of the connections were greylisted away.

Three of the top 30 most rejected IP addresses were rejected 100 times or more; the leader was an internal UofT machine, followed by 80.24.154.168 (135 times) and 64.166.14.222 (134 times). 16 of the top 30 are currently in the CBL and 4 are currently in bl.spamcop.net.

This week's Hotmail numbers:

• 3 messages accepted, at least one of which was spam (I know because I got it).
• 1 message rejected because it came from a non-Hotmail email address (in this case, a msn.com username).
• 41 messages sent to our spamtraps.
• 5 messages refused because their sender addresses had already hit our spamtraps.
• no messages refused due to their origin IP address.

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	2008	175	1780	151
Bad bounces	470	374	482	416

This week's winning bounce target is tosi, with 11 hits, then a couple of old usernames. Plausible usernames (as opposed to the usual random jumbles) seem to be the popular thing this week.

(Next week's bad bounces numbers are probably going to jump some, because I changed how we track them a bit. Up until now, we've only been logging the first bad bounce in a given SMTP session, for reasons that don't fit in this margin; we're now logging all of them.)

Weekly spam summary on November 11th, 2006

This week, we:

• got 15,129 messages from 287 different IP addresses.
• handled 21,714 sessions from 1,659 different IP addresses.
• received 193,764 connections from at least 45,843 different IP addresses.
• hit a highwater of 35 connections being checked at once.

Connection volume is down from last week. The highwater is much higher and was set sometime Thursday; before then we had the same highwater as last week.

Day	Connections	different IPs
Sunday	22,042	+5,058
Monday	26,755	+7,229
Tuesday	25,419	+6,811
Wednesday	29,058	+7,241
Thursday	32,172	+6,803
Friday	31,499	+6,677
Saturday	26,819	+6,024

This shows a Thursday peak as well, ramping up on Wednesday and sliding down on Friday.

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
72.244.103.210        14789    692K
69.31.86.14           11047    663K
203.158.55.63          8565    436K
64.166.14.222          6958    334K
193.252.22.158         5086    305K
212.216.176.0/24       4894    241K
213.1.255.134          4866    234K
210.108.75.241         4186    201K
64.32.177.103          3525    165K
213.4.149.12           3364    175K

It's a miracle: 213.4.149.12 has dropped from consistent first place all the way down to barely ranking this week. Overall it's a bit worse than last week; the high end is worse and the low end is not much better.

• 72.244.103.210 returns from two weeks ago, and is still a Covad something-or-other.
• 69.31.86.14, 193.252.22.158, and 213.4.149.12 all return from last week.
• 203.158.55.63 is an iinet.net.au customer machine in the CBL.
• 64.166.14.222 is a Pacbell DSL line, returning from September.
• 213.1.255.134 and 64.32.177.103 kept dumping bad HELOs on us.
• 210.108.75.241 is a Korean IP address with no reverse DNS.

Just over half of this week's top ten are returning IPs that we've seen before. I tend to find this depressing.

Connection time rejection stats:

  43509 total
  23493 dynamic IP
  16892 bad or no reverse DNS
   1744 class bl-cbl
    221 class bl-sdul
    194 class bl-njabl
    189 class bl-dsbl
     59 class bl-spews
     46 class bl-sbl
     40 class bl-ordb
     35 cuttingedgemedia.com

Two of the top 30 most rejected IP addresses were rejected 100 times or more; 64.166.14.222 (567 times), and an internal UofT client machine that has apparently been misconfigured to try to use us as its server. Fifteen of the top 30 are currently in the CBL and five are currently in bl.spamcop.net.

This week's gifts from Hotmail:

• 2 messages accepted.
• 3 messages rejected because they came from non-Hotmail email addresses.
• 38 messages sent to our spamtraps.
• 2 messages refused because their sender addresses had already hit our spamtraps.
• 1 messages refused due to its origin IP address being in SBL38620 (listed March 4th 2006, and it's apparently an Internet cafe in Nigeria with a satellite Internet connection).

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	1780	151	1298	140
Bad bounces	482	416	370	256

I don't like the upward trend compared to last week, but there's nothing I can do about it.

This week's bad bounce targets are all over the map. Almost nothing got hit more than once ('olsak' is the leader, at 4 times, followed by 'noreply' and a couple of old usernames here at 3 times), and the popularity of Slavic women's first_lastname usernames continues its slide. Apart from that there are words like worm and zodiac, old usernames, vaguely plausible usernames like semerad, a certain amount of capitalized names like Kardel, one XXoX, a lot of random jumbles like zywcfnhiqtji, and the return of a few Linux ALSA function names.

(I have no idea why 'snd_pcm_hw_params_get_buffer_size' is so absurdly popular with spammers as the origin of their forged spam, but it is and has been for years.)

A fundamental problem with challenge/response anti-spam systems

A fundamental problem with CR is that it implicitly assumes that your correspondent wants you to read their email more than you do. Just look at who's doing the work: your correspondents are doing work so that you don't have to.

(Of course, observe that the people for whom this is the most true are the spammers.)

When this is not true, when you want to read the mail more than the sender wants you to, is exactly where CR systems break down. You might think that this is rare, but it's actually quite common; mailing lists are the obvious example. (Trust me; most mailing list managers are completely indifferent about whether the mail reaches you.)

CR proponents like to claim that whitelisting will solve this problem. It doesn't. The fundamental problem of whitelisting is that what you want to whitelist is abstract identities, like 'my friend' or 'my bank', but the only thing available is crude proxies like the email's origin address. And the relationship between the proxies and the real thing changes all the time.

This leads to an important rule:

People who use challenge/response systems should not expect anyone else to expend effort to get their email to them.

Or, the short version: your CR system dropping email is your problem, not mine.

(Disclaimer: and of course CR systems have fundamental problems on the real Internet. There's no way to avoid spamming random bystanders with challenge messages, since almost all spam has forged origin addresses.)

Weekly spam summary on November 4th, 2006

This week, we:

• got 15,265 messages from 278 different IP addresses.
• handled 21,071 sessions from 1,538 different IP addresses.
• received 215,602 connections from at least 52,130 different IP addresses.
• hit a highwater of 12 connections being checked at once.

These statistics superficially look a lot like last week's, although up somewhat. What they hide is a significant spam storm that has actually been getting through our low-rent graylisting, more or less shown in the per day table:

Day	Connections	different IPs
Sunday	26,390	+6,411
Monday	28,908	+7,066
Tuesday	26,318	+6,641
Wednesday	30,790	+7,677
Thursday	33,607	+8,772
Friday	36,044	+8,379
Saturday	33,545	+7,184

Today especially our logs have been lighting up with this stuff. The giveaway sign is dynamic machines HELOing with their actual (dynamic) name, not a forged HELO greeting, and then trying to MAIL FROM various random places. So far most of them have been European IPs, with some Asian and American ones to make life more exciting.

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
213.4.149.12          26060   1355K
69.31.86.14            7864    472K
216.71.64.178          5190    311K
194.213.224.9          5015    301K
209.182.108.85         5014    231K
60.231.152.85          4794    244K
193.252.22.158         3942    237K
61.128.0.0/10          3930    199K
212.216.176.0/24       3917    194K
213.29.7.134           3890    233K

This is a bunch better than last week, with everyone except our usual prize winner coming in significantly lower. Also, almost all of the IPs are new ones.

• 213.4.149.12 and 193.252.22.158 reappear from last week.
• 69.31.86.14 is in SPEWS.
• 216.71.64.178 is a host4u.net machine, and we're not interested in talking to them.
• 194.213.224.9 kept trying to send us stuff that had tripped our spamtraps.
• 209.182.108.85 had a bad HELO greeting.
• 60.231.152.85 is a bigpond.net.au cablemodem. Uh, no thanks.
• 213.29.7.134 is a centrum.cz mail machine (last spotted in February), although a neighboring machine made the list last week.

This is an interestingly broad assortment of reasons for getting blocked, much less monochromatic than usual.

Connection time rejection stats:

  56316 total
  29798 dynamic IP
  21151 bad or no reverse DNS
   3133 class bl-cbl
   1055 class bl-sdul
    261 class bl-dsbl
    115 class bl-njabl
     99 class bl-spews
     48 class bl-ordb
     37 cuttingedgemedia.com
     27 class bl-sbl

And here we see the explosion: this is way up from last week, with major growth in several areas typical of exploited zombie machines.

Four of the top 30 most rejected IP addresses were rejected 100 times or more: 124.90.223.216 (376 times, all of them today), 69.159.193.177 (181 times), 82.163.27.65 (102 times), and 82.3.189.248 (100 times). Annoyingly, one of them is a Canadian (even a Toronto) IP address.

Nineteen of the top 30 most rejected IP addresses as currently in the CBL, and 10 are currently in bl.spamcop.net.

This week's Hotmail grumps:

• 8 messages accepted; one that was good, 6 that were definitely spam, and one I'm not sure about.
• 1 message rejected because it came from a non-Hotmail email address.
• 38 messages sent to our spamtraps.
• 1 message refused because its sender addresses had already hit our spamtraps.
• 6 messages refused due to their origin IP address (3 for being in the CBL, one from SAIX, one from Burkina Faso, and one from the Cote d'Ivoire).

And the final numbers:

what	# this week	(distinct IPs)	# last week	(distinct IPs)
Bad HELOs	1298	140	2076	103
Bad bounces	370	256	377	276

Things are better than last week, but not hugely. The most popular bad bounce target this week was 'wilhelmi' (21 hits), but in general the pattern continued from last week, and almost everything was hit only once. There seems to be a drift towards single-word usernames, away from the combination Slavic female names.

The downsides of remailing

In the context of SPF and its need for SRS instead of simple mail forwarding, a local sysadmin recently asked on our sysadmin mailing list:

What is bad about remailing?

Fundamentally we're being asked to do extra work that benefits other people, people who've chosen to break their own mailers for no actual benefit. This is backwards, and in my opinion accommodating such people only encourages the next lot to demand that everyone else clean up their messes.

Apart from that:

• remailing requires additional software and configuration, especially if you want to stop people from still (accidentally) using non-remailing forwarding.
• simple Unix implementations require people you are remailing for to have something approximating a real account. Forwarding just requires an /etc/aliases entry, which is a lot more reassuringly secure.

• the simplest implementation discards bounces entirely, insuring that if something goes wrong with the forwarding (and things go wrong with forwarding all the time) that no one will ever find out about it.
• slightly more complicated schemes turn you into an open relay if the spammers start forging 'bounces' and sending them through you.
• to make a secure scheme, you either need to keep a database of remailed mail or you run into SMTP address length limits when there is a remailing chain.

(PS: remember to forward the SMTP null origin address unaltered.)

• the origin address is useful information, and for many purposes remailing destroys it. The remote MTA cannot really do filtering or whitelisting on it any more, and people who want to use it in their own filters will have to fish it out of the message headers (with a different fishing technique for every different place things get forwarded to them).

• on a non-technical level, putting your own name on something by remailing it (instead of merely forwarding it) makes you more strongly associated with it. This is a problem when you start remailing spam. It also makes it look more like you really did originate the message, and the other Received: headers are just fakes injected on your machine.

(Obligatory attribution: I mined a bunch of ideas from here and here.)

Sidebar: why SPF is pointless

SPF is pointless because it doesn't solve any actual problems.

• it doesn't stop spam; there are a lot of domains without SPF records that spammers can forge freely, and spammers can and do use their own throwaway domains with valid SPF records.
• it doesn't stop you from getting hammered with bounce backscatter; there are and there's always going to be lots of machines on the Internet that don't implement SPF. (And almost everything that still generates backscatter is well behind the best practices curve to start with.)
• it doesn't stop phishing; the phishers barely bother to forge origin addresses to start with (partly because they're invisible to about 99% of the people reading email).

My experience also suggests that having SPF records doesn't cause spammers to avoid forging your domain.