Wandering Thoughts archives

Our DTrace scripts for NFS server, ZFS, and iSCSI initiator monitoring

As a result of recent events, I've built up a collection of DTrace scripts for monitoring and reporting on our fileserver environment, where we use NFS v3 on top of ZFS on top of iSCSI. Since I grumbled earlier about the lack of easily findable DTrace scripts for this, I've made our scripts public on Github as siebenmann/cks-dtrace (where you can read more details about what's there). They're written for Solaris 10 update 8 (plus some patches) and do play around with kernel data structures.

These scripts are somewhat specific to our environment and contain various local assumptions (some of them commented). They're also not the best DTrace code possible, and in fact they contain several generations of my DTrace code as I steadily learned more about what I was doing (if I was very keen, I would go back to rewrite the older code in the current best style I know).

In addition to their straightforward use, these scripts may serve as a useful example of both how to do various things with DTrace and how to extract various bits of information from the Solaris kernel. In an ideal world there would be a DTrace wiki with information of the form 'given an X, here's how you get a Y' (such as 'given a vnode, here's how you get its ZFS pool'), but as far as I know right now you have to find various little tricks in various people's DTrace scripts.

(I'd be overjoyed to be wrong about this.)

In the 'giving proper credit' department: I didn't come up with these scripts in a void, using only my ingenuity; instead, I stand on the shoulders of many giants. I would not have gotten anywhere near as far I have without taking all sorts of DTrace tricks and clever ways of extracting various bits of kernel information from other people's DTrace scripts, often ZFS-related scripts. Useful references that I have in my notes include Brendan Gregg (and various scripts in the DTrace book's chapter 5) and Richard Elling's zilstat.

(And as always, all of these scripts would have been basically impossible without the OpenSolaris kernel code. That lack of kernel source code cripples DTrace is one reason I remain quite angry with Oracle's decision to close Solaris 11 source. The more I use DTrace, the more convinced I am that we'll never move to Solaris 11 (unless Oracle has a total change of heart).)

Some stats and notes on relay attempts for our external mail gateway

After discovering something attempting some open relay checks, I got curious about whether this was a one-off or if there were clear signs of other open relay checks. To give you a spoiler, the answer is that I can't completely tell because there is a bunch of noise in my data (and on top of that I'm not sure how to analyze it), but it seems possible.

What I can easily get from Exim's logs is triples of IP address, MAIL FROM, and RCPT TO for rejected relay attempts. I have no good way to reconstruct these into sessions, so it's easy to tell someone connecting five times and making a single relay attempt each time apart from someone connecting once and trying a whole series of RCPT TOs.

(I admit that somewhere around here it becomes very tempting to pour all of this data into SQLite and start doing ad hoc queries, because I could really use some GROUP BY clauses right now.)

My raw data covers about 90 days of logs and has 18,290 such triples. These relay attempts come from 1880 different source IPs; out of these, 540 IPs only occur once (so they connected, did a MAIL FROM and a RCPT TO, got a failure, and gave up). Almost all of the origin/destination address pairs are unique (the big exception is test@live.com and its Yahoo destination), but there is a little bit of duplication in RCPT TO addresses (and almost none in MAIL FROMs). At a minimum there appears to be some well-written spam software that immediately gives up if it gets a relaying denied message, rather than try multiple RCPT TOs.

The most active source IPs used multiple MAIL FROMs. For example, the single most active source IP used 23 different MAIL FROMs, almost all of them with multiple RCPT TOs. This I take to be genuine attempts to use us as a relay without particularly noticing (or caring) that none of them work. A few IP addresses tried repeatedly to forge valid local addresses as the MAIL FROMs on their relay attempts, perhaps in an attempt to increase the odds that we'd allow them through; the addresses were all administrative ones like root, info, admin, and so on. It's possible that these were relay probes, because they all seem to have had RCPT TOs of the same addresses (eg, one IP would try a whole bunch of different local MAIL FROMs, all RCPT TO'ing the same remote address). A few people tried the null sender as a MAIL FROM.

(From previous stats I know that spammers forge a lot of bad local usernames on their MAIL FROMs, although that may not be for relay attempts.)

The top destination domains are mostly Asian. Counting only unique would-be recipients (of which there were 17500), the top five domains are:

There were 3104 unique senders and their top five origin domains look sort of similar, but much more evenly distributed:

I think that this is as much random bits and pieces as I want to throw out right now. Part of my problem is that I'm not sure what useful or interesting statistics I can generate from this data, although it feels like there should be something interesting there.