Looking back at DTrace from a Linux eBPF world (some thoughts)

April 26, 2020

As someone who made significant use of locally written DTrace scripts on OmniOS and has since moved from OmniOS to Linux for our current generation of fileservers, I've naturally been watching the growth of Linux's eBPF tooling with significant interest (and some disappointment, since they're still sort of a work in progress). This has left me with some thoughts on the DTrace experience on Solaris and then OmniOS as contrasted with the eBPF experience on Linux.

On Linux, eBPF and the tools surrounding it are a lot more rough and raw than I think DTrace ever was, and certainly than any version of DTrace that I actively used. By the time I started using it, DTrace was basically fully cooked on Solaris and accordingly there was little change between the start of my DTrace experience and the end of it (I think DTrace gained some more convenience functions for things like dealing with IP addresses, but no major shifts). But at the same time, how Linux as a whole has developed eBPF and the tools surrounding it has made Linux eBPF an open system with multiple interfaces (and levels of interface) in a way that DTrace never was, and this has enabled things that at least I never saw people doing on Solaris and Illumos.

In the DTrace world, the interface almost everyone was supposed to use to deal with DTrace was, well, dtrace the command and its language (the DTrace library is explicitly marked as unstable and private in its manual page). You might run the command in various ways and generate programs for it through templates or other things, but that was how you interacted with things. If other levels of interface were documented (such as building raw DTrace programs yourself and feeding them to the kernel), they were definitely not encouraged and as a result I don't think the community ever did much with them.

(People definitely built tools that used the DTrace system that didn't produce processed text output from dtrace, but these were clearly product level work from a dedicated engineer team, not anything you would produce for smaller scale things. Often they came from Sun or a Illumos distribution provider, and so were entitled to use private interfaces.)

By contrast, the Linux eBPF ecology has created a whole suite of tools at various levels of the stack. There's an equivalent of dtrace, but you can also set up eBPF instrumentation of something from inside a huge number of different programming environments and do a bunch of things with the information that it produces. This has led to very flexible things, such as the Cloudflare eBPF Prometheus exporter (which lets you surface Prometheus metrics for anything that you can write an eBPF program for).

I can't help but feel that the Linux eBPF ecology has benefited a lot from the fractured and separated way that eBPF has been developed. No single group owned the entire stack, top to bottom, and so the multiple groups involved were all tacitly forced to provide more or less documented and more or less stable interfaces to each other. The existence of these interfaces then allowed other people to come along and take advantage of them, writing their own tools on top of one or another bit and re-purposing them to do things like create useful Grafana dashboards (via a comment on here). DTrace's single unified development gave us a much more polished dtrace command much sooner (and made it usable on all Solaris versions that supported DTrace the idea), but that was it.

(I don't fault the DTrace developers for keeping libdtrace private and so on; it really is the obvious thing to do in a unitary development environment. Of course you don't want to lock yourself into backward compatibility with a kernel interface or internal implementation that you now realize is not the best idea.)

Written on 26 April 2020.
« Some notes on Firefox's interstitial warning for old TLS versions
My views on SystemTap as compared to eBPF for Linux kernel instrumentation »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Apr 26 01:35:25 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.