2008-07-20
Thinking about uses for (system) activity tracers
System activity tracers are a hot topic, with the best known one being Sun's DTrace. In thinking about this issue recently, I believe that there are three sorts of questions that they can be used to answer, or at least that I'm interested in having answered:
- what is my system doing?
Performance related tracing is one obvious subset of this, both in the 'what is taking all the time' sense and in the 'how long does some operation take' sense.
- why is my system doing X, in the sense of 'what is doing X on my
system'?
Here you have some peculiar thing happening on your system and you want to trace it back to the program or system or action that causes it. For example, laptop people are interested in questions like 'what is accessing my hard drive' and 'what is waking up all the time'.
- why is some part of my system doing what it is, or at least what information is it using to make the decisions about what it does?
The latter is important for solving specific problems; often you know roughly what is going wrong and what program is responsible, but you don't know why and how it is going wrong because you can't see the program's decision making process or even the information it is getting to make the decision. For example, consider 'I can't NFS-mount a filesystem that I think I should be able to'.
In theory you could deal with this by having programs optionally log a lot of information. My personal feeling (partly from having dealt with programs that did copious logging if asked) is that it is better to have a single central interface for deciding what you want to watch and log than to try to give every program options to control all of this; it just scales better, and it's probably easier for program authors too (since they just have to make some hooks available, instead of building a dynamically reconfigurable debug logging system).
Why I'm mostly out of the email (anti-)spam game
I was once fairly interested in and involved in anti-spam stuff; I spent a bunch of time on anti-spam precautions here, followed news sources, and so on. These days, I find myself much less involved, and although I still care about various spam issues, I don't spend very much time involved with the whole field.
What happened is simple: somewhat to my surprise, the spam problem here was pretty much solved when we deployed a commercial anti-spam solution in combination with greylisting and zen.spamhaus.org. It's not perfect, but the amount of spam that got through to me has dropped to almost nothing with almost no effort on my part once we had everything set up.
(One great advantage of commercial solutions is that someone else worries about keeping them up to date. I suspect that the commercial solution is spending far more man-hours than I ever did on this, because this is their speciality and because they can amortize the time over a lot of customers.)
There's still anti-spam improvements I could make to our mail system and I'm still interested in the whole field, but the urgency has gone way down and with it, the amount of time I spend on anti-spam stuff. When the problem seems at least 95% solved, it is hard to carve out the time and motivation to work away at the remaining 5% (especially when there is lots of other work to do).
(I admit that my view is influenced by local attitudes. And I do admit that it feels peculiar and somewhat alarming to delegate something as important as our anti-spam filtering to an outside party, however well it's worked out so far.)