DTrace's stable providers are not good enough

January 4, 2013

In a comment on my entry on why DTrace doesn't attract people to Solaris that often, Brendan Gregg left a comment where he drew a distinction between two levels of using DTrace:

In terms of difficulty, using the DTrace providers is a little like:

  • fbt provider: writing a simple kernel patch
  • stable providers: writing a shell script

Sysadmins should be able to handle stable providers (eg, io, proc, sched, vminfo). They are documented - you don't need to reach for kernel code. Programming them may be no more difficult than shell scripting.

(For 'fbt provider', you should actually read 'any unstable provider' (my DTrace scripts use sdt as well, for example). And I think that using unstable providers is easier than writing a kernel patch in that you can get far with a basic ability to read C code, although my perspective may be skewed.)

I'll start by saying that my experience has given me some strong biases here and it's possible that I'm missing a world of DTrace usage. Also, all of this is from the perspective of someone using Solaris 10 update 8; some of this has changed in Solaris 11 and perhaps with Illumos. With that said, though:

I agree with Brendan's characterization, but the problem is that DTrace's stable providers are not good enough. For a glaring example of this, getting even relatively basic information about NFS server activity requires using unstable providers (although I think this has been fixed in Solaris 11). The cold, hard truth (as I wrote about a bit when I talked about why we hadn't taken to DTrace) is that the Solaris developers never attempted to develop even a vaguely complete set of stable providers. Almost everything really useful is unstable and thus undocumented (and this leads to the need for system programmers and kernel source code in order to write useful DTrace scripts).

At one level, my personal experience is not necessarily representative; we use our Solaris machines as fileservers and have almost no programs running on them locally. If I had to diagnose many local programs I might value the stable providers (my impression is that many of them are at the kernel to userland interface level), but as it stands I don't think I've ever directly used any stable provider and the information from other people's DTrace scripts using them was at most vaguely useful.

One major lack of stable providers is that almost no significant subsystem inside Solaris has stable providers (although Oracle seems to be changing this in Solaris 11, based on documentation). I've mentioned the NFS server and also I think the NFS client, but ZFS is another large example. There are a lot of important and interesting ZFS activities, all of which have to be extracted through unstable providers. Want to watch device multipathing activity to see if everything is fine? Unstable providers again. You get the idea.

(In fact it's not unusual for kstats to provide better visibility into a subsystem than DTrace does with stable providers.)

One effect of unstable providers being so necessary to solve real problems is that the 'shell scripting' level of DTrace is not deeply useful. Sure, you can put together something from documented interfaces and the DTrace manual (if someone hasn't already written the general script and put it on the net), but what you can write probably won't do you much good or tell you things that are very interesting.

Another problem is that relying on unstable providers makes even canned DTrace scripts harder for people to use, as I ran into with my DTrace scripts. Some kernel data structures changed a bit between S10U8 (what we use) and later versions, which means that my scripts don't work as-is for anyone on a later version and need to be edited at least a bit. Requiring sysadmins to make magic edits to scripts before they can use them is not exactly encouraging people to like and use DTrace.

(DTrace doesn't provide any mechanisms to make this easier, although once again it easily could if it actually cared about this issue. But it doesn't, because this is what sysadmins deserve when they use unstable providers, right?)


Comments on this page:

From 86.146.232.75 at 2013-01-04 04:57:55:

Solaris 10 is very old now... as you mentioned yourself there are stable providers for NFS client and server in Solaris 11 (and Open Solaris before). There are also other new stable providers there.

You want to check status of multipathing? I don't know why would you even go to DTrace or similar tool - it is probably better to use tools designed to do so (mpathadm, iostat -X or -Y, fcinfo/fcadm, etc.). Unless you are looking for a bug in MPxIO stack but then a stable provider would most likely not be helpful anyway, and you are talking about dtrace usage from a sysadmin perspective and not a kernel developer.

The point is that while DTrace is a really awesome tool, sometimes it is better and easier to use other tools to answer your question (like multipathing status in your example) instead of developing yet another provider which duplicates functionality which is already there. Do you really want a stable provider to give you uptime or is it better to stick to the old good tool called the same?

Having said that, it doesn't mean DTrace can't be improved on.

Care to provide a couple of examples of real world problems a sysadmin commonly tries to solve, when there is a lack of stable providers? And please, lets stick to Solaris 11 or Illumos and not very old Solaris 10 where only some features got back-ported. Unless you want to talk about history of computing...

By trs80 at 2013-01-05 22:13:46:

I was going to write something about how Linux has problems with stable tracing APIs, but then I came across this post on tracepoints, trace events and perf which then in comments ftrace, SystemTap, LTTng, kprobes, and I realised I still have no idea just what these all do or how they fit together, despite having read LWN articles on the subject.

From 67.188.160.90 at 2013-01-06 15:41:54:

Yes, I realized after posting that I didn't mention stable providers, so I'm glad you brought it up. It's the 3rd strategy for helping people use DTrace: training, tools, stable providers. Different things work for different people. I've taught DTrace to well over a hundred sysadmins (either in a dedicated DTrace class, or introduced during another class), and most of them just wanted the tools. Developers are more willing to code themselves.

The cold, hard truth ... is that the Solaris developers never attempted to develop even a vaguely complete set of stable providers.

Wrong.

DTrace doesn't provide any mechanisms to make this easier, although once again it easily could if it actually cared about this issue.

Easily? I can't guess what other mechanism you have in mind - but it would be great if you have an easy idea for this problem. Can you elaborate?

- Brendan

By cks at 2013-01-07 10:10:58:

Somewhat belated replies:

We don't use Illumos or Solaris 11, so I can't say anything about DTrace on those systems from personal experience. Going only from the Solaris 11 documentation, it appears that there is still no stable provider for ZFS information; this leaves you adrift if, for example, you want to know how much ZIL activity there is to determine if you'd benefit much from a separate SSD log device or if you want to correlate ZIL syncs with other activity.

The Linux tracing situation is clearly worse than DTrace, although people are trying.

Brendan:

The cold, hard truth ... is that the Solaris developers never attempted to develop even a vaguely complete set of stable providers.

Wrong.

My apologies and I'm wrong. I should have said that if they attempted it, they weren't successful.

DTrace doesn't provide any mechanisms to make this easier, although once again it easily could if it actually cared about this issue.

Easily? I can't guess what other mechanism you have in mind - but it would be great if you have an easy idea for this problem. Can you elaborate?

DTrace already knows the shape of the structures involved (the members and their types), so what it needs is some sort of conditional compilation or conditional selection based on what structure members are. Then you could say 'if struct objset has a 'os' member, dereference it like this; otherwise dereference things like that'.

(There are all sorts of more general facilities that I could wish for, like the ability to write subroutines for dereferencing data, but some form of conditional compilation is the starting point.)

Written on 04 January 2013.
« An alterate pattern for polymorphism in C
What I think changed to make spam deliveries not cost-free »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Jan 4 03:35:06 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.