2007-02-18
Why we do NFS fileserving with a SAN
Our storage infrastructure here has a number of NFS servers sitting in front of a pool of SAN RAID storage boxes using commodity SATA disks. This is a somewhat unusual setup for a comparatively small environment like ours; a far more common setup is to have the disks directly attached to the fileservers.
We have a SAN setup for a simple reason: failover between the NFS server machines. We consider the server machines to be the things most likely to suffer failures, either hardware or software, or just to need downtime. With all of the storage in a SAN pool, accessible to any of the frontend machines, we can easily move NFS service from one machine to another.
The actual implementation uses virtual NFS servers and Solaris DiskSuite's failover support, which works quite nicely (although it is not high availability automated failover; we have to kick it off by hand). DiskSuite also lets us mirror important partitions across multiple SAN RAID controllers, so that they'll stay available even if a controller goes down.
It's not clear to me how to set up a similar relatively fast failover environment with directly attached SATA disks. I can think of two approaches; servers with a relatively small number of disks and you just have cold-spare servers waiting, or putting the disks in external shelves and giving all of your NFS servers spare SATA controllers. In either case, 'failover' would be enough work and user disruption that you would be unlikely to use it for things like applying OS patches.
(Then there is the really crazy approach where each server mirrors its local disks over the network to disks on another server via some disk-over-network protocol, whether NBD or iSCSI or the like. The downside is that you need twice as much disk space and either twice as many servers or servers with twice as many drive bays, and in the later case taking down a server means that you lose redundancy on two disk pools, not just one.)
(Credit where credit is due: the crazy approach was suggested by someone at Unix Unanimous.)
Weekly spam summary on February 17th, 2007
This week, we:
- got 15,925 messages from 244 different IP addresses.
- handled 23,465 sessions from 1,341 different IP addresses.
- received 244,268 connections from at least 75,016 different IP addresses.
- hit a highwater of 16 connections being checked at once.
This is about the same as last week. The per day figures show some significant fluctuations:
Day | Connections | different IPs |
Sunday | 36,660 | +13,133 |
Monday | 37,139 | +12,216 |
Tuesday | 43,156 | +12,833 |
Wednesday | 36,296 | +11,682 |
Thursday | 31,349 | +8,987 |
Friday | 32,322 | +8,878 |
Saturday | 27,346 | +7,287 |
Kernel level packet filtering top ten:
Host/Mask Packets Bytes 213.29.7.0/24 14878 892K 64.166.14.222 14215 682K 65.99.209.156 12430 682K 213.4.149.12 9316 484K 68.153.217.220 6508 312K 71.89.4.212 4907 235K 70.246.90.150 4413 212K 66.15.119.165 4186 196K 216.229.180.243 3695 177K 66.42.167.154 3136 150K
This is definitely down from last week, which is welcome, and for the first time in a while 213.4.149.12 (terra.es) is not at the top of the list.
- 64.166.14.222, 213.4.149.12, 68.153.217.220, and 66.15.119.165 all return from last week.
- 65.99.209.156 kept trying to send us spam that had already tripped our spamtraps.
- 71.89.4.212 is a charter.com DHCP machine of some sort.
- 70.246.90.150 kept trying with a bad
HELO
. - 216.229.180.243 kept trying to send what looks like phish spam with
MAIL FROM
s that had already hit our spamtraps. - 66.42.167.154 is in the SORBS DUL.
To my surprise, 208.99.198.64/27 totally disappeared; in contrast to their performance last week, this week we saw not so much as one packet from them. I would like to think that this is because they got disconnected, but I'm not that optimistic.
Connection time rejection stats:
71169 total 44825 dynamic IP 17384 bad or no reverse DNS 6398 class bl-cbl 1004 class bl-sbl 203 class bl-pbl 201 class bl-njabl 183 class bl-sdul 177 class bl-dsbl 81 cuttingedgemedia.com
Almost all of the SBL hits came from 69.42.169.0/24 (914 hits), listed as SBL50892 (spam source and landing pages, listed February 6th) and SBL50451 (colocentral.com spammer hosting, an escalation listing, also listed February 6th). They've showed up before, back in late January, where they were even more active.
(The next highest SBL listing only has 17 rejections; it is SBL49046, a free webmail place listed for (what else) advance fee fraud spamming. After that is SBL50375 (13 rejections, a Rokso-listed place), and SBL50928 (12 rejections, a hijacked server).)
Two out of the top 30 most rejected IP addresses were rejected 100
times or more this week; 64.166.14.222 (631 times) and 60.248.160.38
(109 times). Only 7 out of the top 30 most rejected IP addresses are
currently in the CBL, none are currently in bl.spamcop.net
, and 12
are in the Spamhaus PBL. One
is currently in the SBL: 201.158.98.10 (50 rejections) is in SBL48034, a /21 listing of
'Suavemente LLC', listed February 5th.
This week's Hotmail score is:
- 1 message accepted, almost certainly a legitimate one.
- 3 messages rejected because they came from non-Hotmail email
addresses, all from '
service_banc@msn.com
'. - 34 messages sent to our spamtraps.
- 1 message refused because its sender address had already hit our spamtraps.
- 1 message refused due to its origin IP address being from SAIX aka telkom.co.za.
And the final numbers:
what | # this week | (distinct IPs) | # last week | (distinct IPs) |
Bad HELO s |
979 | 155 | 995 | 154 |
Bad bounces | 9 | 8 | 12 | 8 |
I am amazed; apparently last week's low bad bounces was not just a one-time anomaly. Bad bounces were sent to only 7 different usernames this week, and interestingly all seven of them are accounts that used to exist here. Three bounces went to a relatively current domain name, two bounces went to a somewhat out of date domain name, and four went to an outdated hostname that is a strong spam and spam bounce signature these days.