== Looking back at DTrace from a Linux eBPF world (some thoughts) As someone who made significant use of [[locally written DTrace scripts on OmniOS ZFSDTraceScripts]] and has since moved from OmniOS to Linux for [[our current generation of fileservers ../linux/ZFSFileserverSetupIII]], I've naturally been watching the growth of Linux's eBPF tooling with significant interest (and some disappointment, since [[they're still sort of a work in progress ../linux/EBPFStillInProgress]]). This has left me with some thoughts on the DTrace experience on Solaris and then OmniOS as contrasted with the eBPF experience on Linux. On Linux, eBPF and the tools surrounding it are a lot more rough and raw than I think DTrace ever was, and certainly than any version of DTrace that I actively used. By the time I started using it, DTrace was basically fully cooked on Solaris and accordingly there was little change between the start of my DTrace experience and the end of it (I think DTrace gained some more convenience functions for things like dealing with IP addresses, but no major shifts). But at the same time, how Linux as a whole has developed eBPF and the tools surrounding it has made Linux eBPF an open system with multiple interfaces (and levels of interface) in a way that DTrace never was, and this has enabled things that at least I never saw people doing on Solaris and Illumos. In the DTrace world, the interface almost everyone was supposed to use to deal with DTrace was, well, _dtrace_ the command and its language (the DTrace library is explicitly marked as unstable and private in [[its manual page https://illumos.org/man/3LIB/libdtrace]]). You might run the command in various ways and generate programs for it through templates or other things, but that was how you interacted with things. If other levels of interface were documented (such as building raw DTrace programs yourself and feeding them to the kernel), they were definitely not encouraged and as a result I don't think the community ever did much with them. (People definitely built tools that used the DTrace system that didn't produce processed text output from _dtrace_, but these were clearly product level work from a dedicated engineer team, not anything you would produce for smaller scale things. Often they came from Sun or a Illumos distribution provider, and so were entitled to use private interfaces.) By contrast, the Linux eBPF ecology has created a whole suite of tools at various levels of the stack. There's an equivalent of _dtrace_, but you can also set up eBPF instrumentation of something from inside a huge number of different programming environments and do a bunch of things with the information that it produces. This has led to very flexible things, such as [[the Cloudflare eBPF Prometheus exporter https://github.com/cloudflare/ebpf_exporter]] (which lets you surface Prometheus metrics for anything that you can write an eBPF program for). I can't help but feel that the Linux eBPF ecology has benefited a lot from the fractured and separated way that eBPF has been developed. No single group owned the entire stack, top to bottom, and so the multiple groups involved were all tacitly forced to provide more or less documented and more or less stable interfaces to each other. The existence of these interfaces then allowed other people to come along and take advantage of them, writing their own tools on top of one or another bit and re-purposing them to do things like [[create useful Grafana dashboards https://twitter.com/freebsdfrau/status/1238541870612926465]] (via a comment on [[here ../linux/DisablingUserEBPF]]). DTrace's single unified development gave us a much more polished _dtrace_ command much sooner (and made it usable on all Solaris versions that supported DTrace the idea), but that was it. (I don't fault the DTrace developers for keeping [[libdtrace https://illumos.org/man/3LIB/libdtrace]] private and so on; it really is the obvious thing to do in a unitary development environment. Of course you don't want to lock yourself into backward compatibility with a kernel interface or internal implementation that you now realize is not the best idea.)