Our DTrace scripts for NFS server, ZFS, and iSCSI initiator monitoring
As a result of recent events, I've built up a collection of DTrace scripts for monitoring and reporting on our fileserver environment, where we use NFS v3 on top of ZFS on top of iSCSI. Since I grumbled earlier about the lack of easily findable DTrace scripts for this, I've made our scripts public on Github as siebenmann/cks-dtrace (where you can read more details about what's there). They're written for Solaris 10 update 8 (plus some patches) and do play around with kernel data structures.
These scripts are somewhat specific to our environment and contain various local assumptions (some of them commented). They're also not the best DTrace code possible, and in fact they contain several generations of my DTrace code as I steadily learned more about what I was doing (if I was very keen, I would go back to rewrite the older code in the current best style I know).
In addition to their straightforward use, these scripts may serve as a useful example of both how to do various things with DTrace and how to extract various bits of information from the Solaris kernel. In an ideal world there would be a DTrace wiki with information of the form 'given an X, here's how you get a Y' (such as 'given a vnode, here's how you get its ZFS pool'), but as far as I know right now you have to find various little tricks in various people's DTrace scripts.
(I'd be overjoyed to be wrong about this.)
In the 'giving proper credit' department: I didn't come up with these scripts in a void, using only my ingenuity; instead, I stand on the shoulders of many giants. I would not have gotten anywhere near as far I have without taking all sorts of DTrace tricks and clever ways of extracting various bits of kernel information from other people's DTrace scripts, often ZFS-related scripts. Useful references that I have in my notes include Brendan Gregg (and various scripts in the DTrace book's chapter 5) and Richard Elling's zilstat.
(And as always, all of these scripts would have been basically impossible without the OpenSolaris kernel code. That lack of kernel source code cripples DTrace is one reason I remain quite angry with Oracle's decision to close Solaris 11 source. The more I use DTrace, the more convinced I am that we'll never move to Solaris 11 (unless Oracle has a total change of heart).)
Some stats and notes on relay attempts for our external mail gateway
After discovering something attempting some open relay checks, I got curious about whether this was a one-off or if there were clear signs of other open relay checks. To give you a spoiler, the answer is that I can't completely tell because there is a bunch of noise in my data (and on top of that I'm not sure how to analyze it), but it seems possible.
What I can easily get from Exim's logs is triples of IP address,
RCPT TO for rejected relay attempts. I have no good way to
reconstruct these into sessions, so it's easy to tell someone connecting
five times and making a single relay attempt each time apart from
someone connecting once and trying a whole series of
(I admit that somewhere around here it becomes very tempting to pour
all of this data into SQLite and start doing ad hoc queries, because
I could really use some
GROUP BY clauses right now.)
My raw data covers about 90 days of logs and has 18,290 such triples.
These relay attempts come from 1880 different source IPs; out of
these, 540 IPs only occur once (so they connected, did a
RCPT TO, got a failure, and gave up). Almost all of the
origin/destination address pairs are unique (the big exception is
email@example.com and its Yahoo destination), but there is a little bit of
RCPT TO addresses (and almost none in
MAIL FROMs). At
a minimum there appears to be some well-written spam software that
immediately gives up if it gets a relaying denied message, rather than
The most active source IPs used multiple
MAIL FROMs. For example, the
single most active source IP used 23 different
MAIL FROMs, almost all
of them with multiple
RCPT TOs. This I take to be genuine attempts to
use us as a relay without particularly noticing (or caring) that none
of them work. A few IP addresses tried repeatedly to forge valid local
addresses as the
MAIL FROMs on their relay attempts, perhaps in an
attempt to increase the odds that we'd allow them through; the addresses
were all administrative ones like
admin, and so on.
It's possible that these were relay probes, because they all seem to
RCPT TOs of the same addresses (eg, one IP would try a whole
bunch of different local
MAIL FROMs, all
RCPT TO'ing the same remote
address). A few people tried the null sender as a
(From previous stats I know that
spammers forge a lot of bad local usernames on their
although that may not be for relay attempts.)
The top destination domains are mostly Asian. Counting only unique would-be recipients (of which there were 17500), the top five domains are:
There were 3104 unique senders and their top five origin domains look sort of similar, but much more evenly distributed:
I think that this is as much random bits and pieces as I want to throw out right now. Part of my problem is that I'm not sure what useful or interesting statistics I can generate from this data, although it feels like there should be something interesting there.