== Mdb is so close to being a great tool for introspecting the kernel

The mdb debugger is the standard debugger on Solaris and Illumos
systems (including OmniOS). One very important aspect of _mdb_ is
that it has a lot of support for kernel 'debugging', which for
ordinary people actually means 'getting detailed status information
out of the kernel'. For instance, if you want to know a great deal
about where your kernel memory is going you're going to want the
'_::kmastat_' mdb command.

Mdb is capable of some [[very powerful tricks ListingFileLocks]]
because it lets you compose its commands together in 'pipelines'.
Mdb has a large selection of things to report information (like
the aforementioned _::kmastat_) and things to let you build your
own pipelines (eg walkers and _::print_). All of this is great,
and far better than what most other systems have.
Where mdb sadly falls down is that this is all it has; it has no
scripting or programming language. This puts an unfortunate hard
upper bound on what you can extract from the kernel via _mdb_ without
a huge amount of post-processing on your part. For instance, as far
as I know a pipeline can't have conditions or filtering so that you
further process only selected items that one stage of a pipeline
produces. In the case of [[listing file locks ListingFileLocks]],
you're out of luck if you want to work on only selected files instead
of all of them.

I understand (I think) where this limitation comes from. Part of
it is probably simply the era _mdb_ was written in (which was not
yet a time when people shoved extension languages into everything
that moved), and part of it is likely that the code of _mdb_ is
also much of the code of the embedded kernel debugger _kmdb_. But
from my perspective it's also a big missed opportunity. A _mdb_
with scripting would let you filter pipelines and write your own
powerful information dumping and object traversal commands,
significantly extending the scope of what you could conveniently
extract from the kernel. And the presence of pipelines in _mdb_
show that its creators were quite aware of the power of flexibly
processing and recombining things in a debugger.

(Custom scripting also has obvious uses for debugging user level
programs, where a complex program may be full of its own idioms and
data structures that cry out for the equivalent of kernel dcmds and
walkers.)

PS: Technically you can extend _mdb_ by writing new mdb modules in
C, since they're just _.so_s that are loaded dynamically; there's
even a [[more or less documented module API
https://illumos.org/books/mdb/api-5.html]]. In practice my reaction
is 'good luck with that'.