Notes on Linux's blktrace
Blktrace is a tool that captures detailed traces of what's happening
in the kernel's block layer. In other words, think of it as a
tcpdump for disk IO. This is just the kind of thing you need to
deal with the fact that averages are misleading; since blktrace can tell you about the
timing of every request, you can use it as the starting point for
all sorts of detailed analysis. Or you can simply use it to verify
that you don't have any timing outliers in your disk IO. Blktrace
requires that your kernel be vaguely modern and have support for
block tracing enabled, but both of these are routine. I think that
most distributions have prepackaged versions of it that are just a
package manager command away; if not, see here.
Blktrace needs no special support or configuration apart from having
debugfs mounted on
/sys/kernel/debug, and if you don't have that
blktrace will tell you so you can fix it with a '
mount -t debugfs
My brief recent
blktrace have all been positive. It just works
and it doesn't seem to demand anything much from the system. I felt
comfortable applying it to a production machine and I didn't observe
any effects of doing so.
Here are some scattered notes on it:
See also this blog entry on it from 2009
I may have more notes and commentary later, when the dust around here
has settled and I've explored blktrace more. But on the positive side,
it was very easy to use blktrace and blkparse to verify that our
problem was not with the disks and being masked by an average.
(Never let it be said that I don't take requests. In fact I
love requests since they mean I don't have to come up with entry