Wandering Thoughts archives

Exploring a limitation of the DTrace fbt provider (on x86)

The fine documentation for DTrace's kernel function tracer contains a little note that you might easily miss if, like me, you were just skimming:

For each instruction set, there are a small number of functions that do not call other functions and are highly optimized by the compiler (so-called leaf functions) that cannot be instrumented by FBT.

The first thing to know is that DTrace does not have any useful error messages when you run into this. Instead of a helpful note about, say, 'this function exists but cannot be traced', you get the generic report 'probe description fbt:<X>:entry does not match any probes'. If you know that the function exists, this is what you're running into.

(And speaking from personal experience, it is very puzzling and annoying if you've run into it and you didn't notice the note in the documentation. I spent rather a lot of time beating my head against this.)

The 'x86 Limitations' section of the fbt documentation has slightly more detail about what is going on; the real condition is that any function that doesn't create a stack frame can't be traced. 'Highly optimized' is a little bit of a misnomer, as I suspect that the real condition is closer to 'doesn't do very much'.

(In the case that I ran into yesterday, the function just did some very simple math with some global variables in order to return a boolean yes/no value.)

Why this limitation exists is that DTrace's fbt provider works by overwriting a bit of the function prequel so that it invokes DTrace hooks instead. Of course the DTrace hooks then have to do what the overwritten code would have done, and this only works if they know what that is and can easily duplicate it. On x86, the interception is done in part of the standard instruction sequence that sets up the function's stack frame; a function with no stack frame has no standard instruction sequence to set it up and thus no place for DTrace to safely overwrite.

Specifically, on 64-bit x86 DTrace overwrites pushq %rbp with int $0x3. You can actually see this in action with mdb -k if you want to by disassembling the first instruction of the function (with 'func/i') before and after you add a DTrace probe on the function. Note that fbt instruments both function entry and function returns no matter which you asked for.

This provides a convenient way to both check to see if a function exists and if it can be traced by fbt; start up mdb -k and disassemble the start of the function, looking for the pushq. If mdb can't find the function at all, there's a difference between the OpenSolaris code you're looking at and the Solaris that you're running.

(More details about this are in uts/intel/dtrace/fbt.c in the OpenSolaris code.)

PS: I'm aware that using 'mdb -k' instead of just having DTrace dump all fbt hooks is only convenient for a certain sort of person. I just happen to be that sort of person.

My view of OpenSolaris and Illumos

My perspective is that I have always had a two-fold interest in OpenSolaris. One part of my interest was actually in Solaris source code (for which OpenSolaris was the closest available substitute), and the other part was an alternative to Solaris if Sun went crazy.

I never truly trusted OpenSolaris for two reasons. First, because I never fully trusted the depth of Sun's commitment to open source; it always felt like a tactical move, instead of a heartfelt conviction, and tactical moves are subject to revocation. Second, OpenSolaris as a usable operating system (ie as an alternative to Solaris) always relied on Sun's charity, because it required closed source binary modules in order to do useful and vital things like NFS locking (and it required Sun compilers in order to build). If Sun had ever wanted to kill OpenSolaris as an operating system, all it ever had to do was stop making those components available.

In short, in effect OpenSolaris was a BSD Net/2 release that happened to run on some hardware. It was the starting point for an open source operating system, but it was not an open source operating system by itself. In practice it was little more than bleeding edge Solaris that you got most of the source code for (cf an earlier entry).

Of course, in theory all of this could change (or could have changed); given the existing source code, the community could fork OpenSolaris and turn it into a real open source OS. But as long as Sun supported OpenSolaris this was never going to happen, because it was too much work for too little gain (and too much difficulty), and so the fundamental uncertainties of OpenSolaris were not going to go away.

In this respect, Oracle did OpenSolaris a favour by throwing it off the back of the truck and forcing people to create Illumos from the wreckage. It is far from certain that Illumos will succeed, but at least the ambiguity of the situation is gone; within not too much time either we will have a viable independent alternative to Solaris or it will be clear that Illumos cannot deliver and is dead in practice (regardless of what people say). As a result I feel much more positive about Illumos than I ever did about OpenSolaris.

Another way to put this is that I never felt Sun was committed to OpenSolaris. People seem clearly committed to Illumos and they have significant money on the line as a motive to make it work, so it is now much more likely that we will wind up with a genuine alternative to Solaris.