Looking back at DTrace from a Linux eBPF world (some thoughts)
As someone who made significant use of locally written DTrace scripts on OmniOS and has since moved from OmniOS to Linux for our current generation of fileservers, I've naturally been watching the growth of Linux's eBPF tooling with significant interest (and some disappointment, since they're still sort of a work in progress). This has left me with some thoughts on the DTrace experience on Solaris and then OmniOS as contrasted with the eBPF experience on Linux.
On Linux, eBPF and the tools surrounding it are a lot more rough and raw than I think DTrace ever was, and certainly than any version of DTrace that I actively used. By the time I started using it, DTrace was basically fully cooked on Solaris and accordingly there was little change between the start of my DTrace experience and the end of it (I think DTrace gained some more convenience functions for things like dealing with IP addresses, but no major shifts). But at the same time, how Linux as a whole has developed eBPF and the tools surrounding it has made Linux eBPF an open system with multiple interfaces (and levels of interface) in a way that DTrace never was, and this has enabled things that at least I never saw people doing on Solaris and Illumos.
In the DTrace world, the interface almost everyone was supposed to
use to deal with DTrace was, well,
dtrace the command and its
language (the DTrace library is explicitly marked as unstable and
private in its manual page).
You might run the command in various ways and generate programs for
it through templates or other things, but that was how you interacted
with things. If other levels of interface were documented (such as
building raw DTrace programs yourself and feeding them to the
kernel), they were definitely not encouraged and as a result I don't
think the community ever did much with them.
(People definitely built tools that used the DTrace system that
didn't produce processed text output from
dtrace, but these were
clearly product level work from a dedicated engineer team, not
anything you would produce for smaller scale things. Often they
came from Sun or a Illumos distribution provider, and so were
entitled to use private interfaces.)
By contrast, the Linux eBPF ecology has created a whole suite of
tools at various levels of the stack. There's an equivalent of
dtrace, but you can also set up eBPF instrumentation of something
from inside a huge number of different programming environments and
do a bunch of things with the information that it produces. This
has led to very flexible things, such as the Cloudflare eBPF
(which lets you surface Prometheus metrics for anything that you
can write an eBPF program for).
I can't help but feel that the Linux eBPF ecology has benefited a
lot from the fractured and separated way that eBPF has been developed.
No single group owned the entire stack, top to bottom, and so the
multiple groups involved were all tacitly forced to provide more
or less documented and more or less stable interfaces to each other.
The existence of these interfaces then allowed other people to come
along and take advantage of them, writing their own tools on top
of one or another bit and re-purposing them to do things like
create useful Grafana dashboards (via
a comment on here). DTrace's single
unified development gave us a much more polished
much sooner (and made it usable on all Solaris versions that supported
DTrace the idea), but that was it.
(I don't fault the DTrace developers for keeping libdtrace private and so on; it really is the obvious thing to do in a unitary development environment. Of course you don't want to lock yourself into backward compatibility with a kernel interface or internal implementation that you now realize is not the best idea.)