Our DTrace scripts for NFS server, ZFS, and iSCSI initiator monitoring
October 31, 2012
As a result of recent events, I've built up a collection of DTrace scripts for monitoring and reporting on our fileserver environment, where we use NFS v3 on top of ZFS on top of iSCSI. Since I grumbled earlier about the lack of easily findable DTrace scripts for this, I've made our scripts public on Github as siebenmann/cks-dtrace (where you can read more details about what's there). They're written for Solaris 10 update 8 (plus some patches) and do play around with kernel data structures.
These scripts are somewhat specific to our environment and contain various local assumptions (some of them commented). They're also not the best DTrace code possible, and in fact they contain several generations of my DTrace code as I steadily learned more about what I was doing (if I was very keen, I would go back to rewrite the older code in the current best style I know).
In addition to their straightforward use, these scripts may serve as a useful example of both how to do various things with DTrace and how to extract various bits of information from the Solaris kernel. In an ideal world there would be a DTrace wiki with information of the form 'given an X, here's how you get a Y' (such as 'given a vnode, here's how you get its ZFS pool'), but as far as I know right now you have to find various little tricks in various people's DTrace scripts.
(I'd be overjoyed to be wrong about this.)
In the 'giving proper credit' department: I didn't come up with these scripts in a void, using only my ingenuity; instead, I stand on the shoulders of many giants. I would not have gotten anywhere near as far I have without taking all sorts of DTrace tricks and clever ways of extracting various bits of kernel information from other people's DTrace scripts, often ZFS-related scripts. Useful references that I have in my notes include Brendan Gregg (and various scripts in the DTrace book's chapter 5) and Richard Elling's zilstat.
(And as always, all of these scripts would have been basically impossible without the OpenSolaris kernel code. That lack of kernel source code cripples DTrace is one reason I remain quite angry with Oracle's decision to close Solaris 11 source. The more I use DTrace, the more convinced I am that we'll never move to Solaris 11 (unless Oracle has a total change of heart).)
Written on 31 October 2012.
* * *