My views on SystemTap as compared to eBPF for Linux kernel instrumentation
In the small lobste.rs discussion on yesterday's entry looking back at DTrace from an eBPF world, jmtd noticed that there was no mention of SystemTap in my entry. SystemTap (also) was once the promising rough Linux equivalent of DTrace, but it's fallen out of favour. I used SystemTap a very little bit back when it was new (for example, to trace the system calls of a setuid program) but did nothing with it since then.
As a sysadmin, I felt that SystemTap had two or perhaps three significant issues. The lesser and questionable issue was that SystemTap was hard to get started with; getting all of the packages on to your system was a little bit intricate in the beginning and the documentation was discouragingly spotty both initially and somewhat later on when I tried it again in 2013. The big issues were that SystemTap was quite slow to start doing things and worked by the scary method of building and loading a new Linux kernel module into the kernel; I believe the kernel module then live-patched the running kernel code on the fly at least some of the time.
These two issues were interrelated; SystemTap was slow because it had to do a great deal of work to create and load that kernel module, and part of why it had to be slow was that a loadable kernel module is very dangerous, and SystemTap was trying to do it without blowing up your system. Because SystemTap used loadable kernel modules, any bugs in its runtime or how it translated your SystemTap script to kernel code could potentially crash your system, which was a scary thing to have to worry about on a production system. The very last thing you want to risk with an instrumentation system is that it makes your problems worse.
All of this effectively made SystemTap a last ditch option for us, something that we would have turned to only if a problem got really bad and it was worth risking crashing a production server in an attempt to diagnose it. In practice our problems never got that bad; we were always able to deal with them in less risky ways.
My perception of eBPF so far is that it is far more stable and officially supported. What eBPF can do is deliberately limited, as compared to the unlimited power of a loadable kernel module, and the core kernel itself works to make eBPF limited and safe. This includes otherwise potentially risky things like hooking into kernel functions with kprobes. This limited power and limited capability for damage makes me much more willing to use eBPF on production systems (once it gets fully available, which is not before Ubuntu 20.04 at the earliest for us). We'll have to test it, of course, but I consider eBPF to be much more intrinsically safe than a loadable kernel module that is supposed to not have any bugs.
(eBPF has had security bugs, which is why we've turned off user access to it, but I don't think it's had many stability issues. Partly I believe this is because its safety gets a lot of attention, since it's part of the core kernel and increasingly heavily used.)