== Using SystemTap to trace the system calls of setuid programs on Linux

Suppose that you have a setuid program that is failing mysteriously
and you want to see what it's doing.  With normal programs you can use
_strace_, but not even root can _strace_ a setuid program (if you try,
the program runs non-setuid).

(Yes, _strace_ has the _-u_ option, but it doesn't help if the setuid
program is being run as part of a whole chain of processes in a specific
environment and you can't just run it directly. It would be nice if root
could use '_strace -f ..._' for this, but alas it doesn't work.)

On a Solaris system you could use DTrace for this. [[SystemTap
http://sourceware.org/systemtap/]] is the rough Linux equivalent and,
although much less polished and not as well documented, it does work.
Here is the crude SystemTap script that I used:

 probe syscall.* {
   en = execname();
   ui = uid();
   eui = euid();
   if (en == "<redacted>") {
     printf("%s(%d): %s(%s)", en, pid(), name, argstr);
     if (ui != eui) {
       printf(" as %d/%d ", ui, eui);
     } else {
       printf(" as %d ", ui);
     }
   }
 }
 
 probe syscall.*.return {
   en = execname();
   if (en == "<redacted>") {
     printf("= %s\n", retstr);
   }
 }

This produces output with system call arguments and return values
helpfully decoded for you; it looks like:

> ((<redacted>(14087): open("/etc/passwd", O_RDONLY) as 2315/0 = 3)) \\
> [...] \\
> ((<redacted>(14087): close(1) as 2315/0 = -9 (EBADF)))

(In some ways this is nicer than DTrace. But the lack of documentation
on what sort of information you can get about system calls and so on
really hurts; I had to read the source for the syscall tapset in order
to find out about _name_, _argstr_, _retstr_, and so on.)

Note that, despite the presence of the PID in the output, this isn't
really useful for tracing if more than one instance of the program is
running at once. That would take more SystemTap magic than I know so far
(or worse output and some postprocessing).  Also, since _stap_ is kind
of slow you'll want to run it with the _-v_ flag so that you know when
it's actually finished checking, compiling, and enabling your tracing.

One of the things that the documentation isn't very clear about is that
the _execname()_ function returns the bare command name of the current
process and not its full path. (There is probably a way to extract the
full path if you need it. I didn't, so I didn't go digging.)

All in all, I would have to score my first real exposure to
[[SystemTap]] as a reasonably pleasant experience. Although there were
a bunch of frustrating bits, it did work, it gave me what I wanted to
know, and it wasn't particularly difficult to do or to work out how to
do it (and it didn't take particularly long).