I've finally gotten around to trying out DTrace, and I have to say that it actually is pretty cool so far. I haven't used it on kernel side stuff, just to poke at user programs, for which it makes a nice print based debugger; it's easy to point DTrace at a process and see what's going on (easier than attaching a debugger and getting anywhere in my opinion), and you have a bunch of interesting analysis options, none of which require you to sit there holding the debugger's hand.

(For example, it is easy to see all of the library calls that a program makes, or all of the calls that it makes to a specific shared library; this is a powerful way of working out how a program is making decisions.)

One drawback to using DTrace on userland programs is that DTrace is fundamentally a kernel debugger, so it does not give you direct access to a process's user-side memory and thus its variables and data structures. In order to get any of this sort of thing, you have to first copy it from user space with various DTrace functions, primarily copyin() for random data and copyinstr() for C-style strings. Another drawback is that DTrace has no looping and very little conditional decision making, which makes it hard to trace down complex data structures.

(I understand why DTrace has to have this basic model, but I really wish that the DTrace people had put more convenience functions in the DTrace language for this. And I don't understand the whole avoidance of an if or equivalent at all.)

That DTrace is not a user-level debugger is actually reassuring in one sense; user-level debuggers are traditionally rather invasive and do things like stopping the process you're monitoring while you hold their hand. This is alarming for a sysadmin, since the processes we want to monitor are often rather important system ones that we want disturbed as little as possible (and certainly not stopped and potentially aborted).