Mdb is so close to being a great tool for introspecting the kernel

July 15, 2015

The mdb debugger is the standard debugger on Solaris and Illumos systems (including OmniOS). One very important aspect of mdb is that it has a lot of support for kernel 'debugging', which for ordinary people actually means 'getting detailed status information out of the kernel'. For instance, if you want to know a great deal about where your kernel memory is going you're going to want the '::kmastat' mdb command.

Mdb is capable of some very powerful tricks because it lets you compose its commands together in 'pipelines'. Mdb has a large selection of things to report information (like the aforementioned ::kmastat) and things to let you build your own pipelines (eg walkers and ::print). All of this is great, and far better than what most other systems have. Where mdb sadly falls down is that this is all it has; it has no scripting or programming language. This puts an unfortunate hard upper bound on what you can extract from the kernel via mdb without a huge amount of post-processing on your part. For instance, as far as I know a pipeline can't have conditions or filtering so that you further process only selected items that one stage of a pipeline produces. In the case of listing file locks, you're out of luck if you want to work on only selected files instead of all of them.

I understand (I think) where this limitation comes from. Part of it is probably simply the era mdb was written in (which was not yet a time when people shoved extension languages into everything that moved), and part of it is likely that the code of mdb is also much of the code of the embedded kernel debugger kmdb. But from my perspective it's also a big missed opportunity. A mdb with scripting would let you filter pipelines and write your own powerful information dumping and object traversal commands, significantly extending the scope of what you could conveniently extract from the kernel. And the presence of pipelines in mdb show that its creators were quite aware of the power of flexibly processing and recombining things in a debugger.

(Custom scripting also has obvious uses for debugging user level programs, where a complex program may be full of its own idioms and data structures that cry out for the equivalent of kernel dcmds and walkers.)

PS: Technically you can extend mdb by writing new mdb modules in C, since they're just .sos that are loaded dynamically; there's even a more or less documented module API. In practice my reaction is 'good luck with that'.

Comments on this page:

You're basically correct that the development of mdb (which started in 1997) essentially predates the rise of scripting languages. It's also true that we found that elaborate pipelines took the place of scripts (if you haven't discovered it, using "!" to pipe mdb output into the shell is very useful; between that and "::cat", you can use all of Unix to massage output). That said, I would love to add (say) a Lua interpreter to mdb; if you make it to an illumos event (e.g., illumos day at Surge?), this is something that I would love to hack on...

Written on 15 July 2015.
« Don't support shortened versions of your domain names in email addresses
Eating memory in Python the easy way »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Jul 15 02:50:04 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.