Applying low distraction design to alerting systems

June 21, 2010

Writing yesterday's entry has left me with some thoughts on creating low-distraction alerting and monitoring systems. Obviously this should only include informative monitoring, but once you've got that you still need to present the information on what alerts are active in a good way. And because you want sysadmins to check your alerts page relatively frequently, you want it to be low distraction in the same way that email checks should be.

A low distraction system needs to show you enough information for you to make at least a preliminary decision, present events in some useful order, and let you shut it up. So, what I think you want is:

  • a display that is organized by severity of alarm and reverse chronological order within that, with the most recent alarm on top and thus the most visible, with either the ages or the start times shown.

  • some sort of one-line summary of each alarm's specific details, so that you don't have to drill down further to find out what the actual problem is.

  • a way of hiding or dismissing specific alarms. Probably you should have a way of canceling this and re-revealing all current alarms.

For an added bonus, default to aggregating alarms together in some way if they are chronologically close enough (with an option to expand out the full details). This provides a natural way to condense cascade failures down into a single alert, crudely solving the alerting dependency problem.

Intuitively, I think that by priority and then chronologically is the right order to sort events into. In most situations I care more about a recent issue than an older one (after all, if the systems haven't entirely melted down by now the older issue can probably wait a bit longer), and more about an older high priority issue than a newer lower priority one. This is arguable and may depend on local circumstances.

(And the priorities may involve things like 'what machine is this reported on', with some machines being much more important than others.)

Written on 21 June 2010.
« How my mail notifier avoids interrupting me
Why feed readers are not good for skimming things »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jun 21 00:27:48 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.