Wandering Thoughts archives

Why we do NFS fileserving with a SAN

Our storage infrastructure here has a number of NFS servers sitting in front of a pool of SAN RAID storage boxes using commodity SATA disks. This is a somewhat unusual setup for a comparatively small environment like ours; a far more common setup is to have the disks directly attached to the fileservers.

We have a SAN setup for a simple reason: failover between the NFS server machines. We consider the server machines to be the things most likely to suffer failures, either hardware or software, or just to need downtime. With all of the storage in a SAN pool, accessible to any of the frontend machines, we can easily move NFS service from one machine to another.

The actual implementation uses virtual NFS servers and Solaris DiskSuite's failover support, which works quite nicely (although it is not high availability automated failover; we have to kick it off by hand). DiskSuite also lets us mirror important partitions across multiple SAN RAID controllers, so that they'll stay available even if a controller goes down.

It's not clear to me how to set up a similar relatively fast failover environment with directly attached SATA disks. I can think of two approaches; servers with a relatively small number of disks and you just have cold-spare servers waiting, or putting the disks in external shelves and giving all of your NFS servers spare SATA controllers. In either case, 'failover' would be enough work and user disruption that you would be unlikely to use it for things like applying OS patches.

(Then there is the really crazy approach where each server mirrors its local disks over the network to disks on another server via some disk-over-network protocol, whether NBD or iSCSI or the like. The downside is that you need twice as much disk space and either twice as many servers or servers with twice as many drive bays, and in the later case taking down a server means that you lose redundancy on two disk pools, not just one.)

(Credit where credit is due: the crazy approach was suggested by someone at Unix Unanimous.)

Weekly spam summary on February 17th, 2007

This week, we:

• got 15,925 messages from 244 different IP addresses.
• handled 23,465 sessions from 1,341 different IP addresses.
• received 244,268 connections from at least 75,016 different IP addresses.
• hit a highwater of 16 connections being checked at once.

This is about the same as last week. The per day figures show some significant fluctuations:

Kernel level packet filtering top ten:

Host/Mask           Packets   Bytes
213.29.7.0/24         14878    892K
64.166.14.222         14215    682K
65.99.209.156         12430    682K
213.4.149.12           9316    484K
68.153.217.220         6508    312K
71.89.4.212            4907    235K
70.246.90.150          4413    212K
66.15.119.165          4186    196K
216.229.180.243        3695    177K
66.42.167.154          3136    150K

This is definitely down from last week, which is welcome, and for the first time in a while 213.4.149.12 (terra.es) is not at the top of the list.

• 64.166.14.222, 213.4.149.12, 68.153.217.220, and 66.15.119.165 all return from last week.
• 65.99.209.156 kept trying to send us spam that had already tripped our spamtraps.
• 71.89.4.212 is a charter.com DHCP machine of some sort.
• 70.246.90.150 kept trying with a bad HELO.
• 216.229.180.243 kept trying to send what looks like phish spam with MAIL FROMs that had already hit our spamtraps.
• 66.42.167.154 is in the SORBS DUL.

To my surprise, 208.99.198.64/27 totally disappeared; in contrast to their performance last week, this week we saw not so much as one packet from them. I would like to think that this is because they got disconnected, but I'm not that optimistic.

Connection time rejection stats:

  71169 total
  44825 dynamic IP
  17384 bad or no reverse DNS
   6398 class bl-cbl
   1004 class bl-sbl
    203 class bl-pbl
    201 class bl-njabl
    183 class bl-sdul
    177 class bl-dsbl
     81 cuttingedgemedia.com

Almost all of the SBL hits came from 69.42.169.0/24 (914 hits), listed as SBL50892 (spam source and landing pages, listed February 6th) and SBL50451 (colocentral.com spammer hosting, an escalation listing, also listed February 6th). They've showed up before, back in late January, where they were even more active.

(The next highest SBL listing only has 17 rejections; it is SBL49046, a free webmail place listed for (what else) advance fee fraud spamming. After that is SBL50375 (13 rejections, a Rokso-listed place), and SBL50928 (12 rejections, a hijacked server).)

Two out of the top 30 most rejected IP addresses were rejected 100 times or more this week; 64.166.14.222 (631 times) and 60.248.160.38 (109 times). Only 7 out of the top 30 most rejected IP addresses are currently in the CBL, none are currently in bl.spamcop.net, and 12 are in the Spamhaus PBL. One is currently in the SBL: 201.158.98.10 (50 rejections) is in SBL48034, a /21 listing of 'Suavemente LLC', listed February 5th.

This week's Hotmail score is:

• 1 message accepted, almost certainly a legitimate one.
• 3 messages rejected because they came from non-Hotmail email addresses, all from 'service_banc@msn.com'.
• 34 messages sent to our spamtraps.
• 1 message refused because its sender address had already hit our spamtraps.
• 1 message refused due to its origin IP address being from SAIX aka telkom.co.za.

And the final numbers:

I am amazed; apparently last week's low bad bounces was not just a one-time anomaly. Bad bounces were sent to only 7 different usernames this week, and interestingly all seven of them are accounts that used to exist here. Three bounces went to a relatively current domain name, two bounces went to a somewhat out of date domain name, and four went to an outdated hostname that is a strong spam and spam bounce signature these days.