Metrics considered dangerous

June 17, 2006

It's a general truism in software development that metrics are often dangerous, because people will naturally work to what you pay attention to (to summarize). If you pay attention to bugs closed, people close bugs; if you pay attention to lines of code written, people write code. The dangers inherent in this are hopefully obvious.

(In fact it happens outside of software development too; consider all of the schools that are busy 'teaching to the test', because various standardized tests have become so important.)

I've had the interesting experience of discovering that this effect works even on yourself, with personal metrics you do for your own use.

A while back I wrote a command to show a count of WanderingThoughts entries broken down by the category and the month they were posted in, because I was curious what the distribution would look like. But once I had this metric, I found myself looking at it and having it affect what I wrote about and when, especially towards the ends of the month as I ran out of time to 'even up' the month's numbers or write as many extra entries as the previous month. (Partly this is my own neurotic nature at work, of course.)

This isn't really how I want to write WanderingThoughts. While a variety of topics is nice, I don't want a drive for variety (and having 'enough' of each topic) to trump writing about whatever catches my fancy. (Down that route lies writing entries just because I need to make the numbers come out, and shortly after that comes doom.)

Since I've become conscious of this effect, I've tried to take steps against it. One of them is obvious: look at the metrics (ie, run the command) less often (although it remains a shiny temptation). Still, I haven't entirely been able to ignore the feeling that I am (for example) writing 'too little' about Python this month.

It's been a somewhat disconcerting to actually live out the truism myself, but it has given me a rather more visceral appreciation for its essential truth. And for how it can sneak up on you, despite the best of intentions.

Written on 17 June 2006.
« /etc/inittab versus /etc/rc.d
Weekly spam summary on June 17th, 2006 »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Jun 17 01:55:27 2006
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.