2011-10-21
How I'm capturing only the last portion of standard error
As part of my migration to Fedora 15, I have been dealing with a buggy program that crashes a lot. It has an option to dump lots of debugging information as it runs and collect stack backtraces when it crashes, and of course the developers want you to do all of this when you report problems to them. Inconveniently, right after I installed all of the debugging packages necessary for good stack backtraces the program seems to have stopped crashing. For now. I don't trust it and when it crashes next I certainly want to capture everything necessary for a good bug report.
(Since the program uses threads, I suspect that it has races that are now being masked by it slowing down to print all of that debugging output.)
Now, there's a problem: this is a program that I want to leave running all of the time. All of that debugging information adds up very fast; I was looking at many tens of megabytes of output a day, most of which was going to be pointless (when the program crashes the developers are only going to want to the last bits of the debug logs). What I wanted to do was only keep the last so much of the program's debugging output, not all of it.
This is of course the general issue of log rotation and standard error. I've written about this before but now I actually needed a program to deal with the problem, something that would capture and rotate the program's ongoing log messages to keep only the last so many of them. Looking back now, the first comment on that entry has a useful index of tools for doing this, but at the time a couple of days ago when I needed this I didn't remember the entry so I reached for the first tool I had at hand that I knew could do the job: djb's multilog.
I have multilog sitting around because I am still using dnscache as my local caching DNS resolver
for various reasons (I'm planning to switch
to unbound at some point, but configuring it for my setup is a huge
pain). Multilog has many features and is total overkill for this
specific need, but I knew I could make it go and that counts for a lot
when a sysadmin is in a hurry.
To save myself (and anyone else) the trouble of sorting through the multilog manpage, here is the command line you want:
program | multilog '+*' sBYTES /tmp/where
/tmp/where must be a directory. BYTES is the size of individual log
files; multilog will keep ten of them (you can change that by adding
nNUM before the directory). The '+*' just tells multilog to
put all log messages it sees in the log it keeps in /tmp/where. Since
I have the disk space I'm using 1 megabyte log files with the default
ten of them.
There's very little to say about the end result. It works. I wrapped the whole invocation of the crashing program up in a little script so that I can forget about the whole thing, at least until it crashes. (And if it never crashes, well, that's what I wanted in the first place; ten megabytes of rotating logs is a minor price to pay for it.)