2005-07-17
Skills I use when troubleshooting
A while back I wrote about FutureSysadminJobs and suggested that people who wanted satisfying careers as system administrators over the long term should develop the skills to be troubleshooters. Which begs the question: what are those skills?
The first and most important skill is: you have to find all this interesting. Enjoying being a curious packrat is not entirely required, but I think it helps a whole lot, especially as you'll probably need to learn a number of things on your own time.
Other than that, based on my own experiences troubleshooting various issues I'm going to say:
Troubleshooters have to know how to program. Really program, not just write little scripts. Sometimes you'll write programs and sometimes you'll have to understand them, and you can't do either if you can't program yourself. (This is probably an unpopular view, but I feel that anyone who can't program is fundamentally crippled here.)
Troubleshooters have to know how to debug, which is harder than it looks. Debugging is part instincts and part paranoia and part obsessive completeness and almost entirely without useful textbooks, which means you have to learn it the hard way, by doing it.
Troubleshooters have to know how things work, because if you don't understand how things work you can't see where they can go wrong. (This means that you are going to be storing away a lot of trivia in your mind. It will help a lot if you like doing this.)
Troubleshooters need to know how to dive into big programs, zoom right in on the one little relevant bit, understand it, and then change it. This is a distinctly different skill than normal program maintenance, and like debugging you mostly get to develop it by being thrown in the deep end.
Similarly, you need to be able to dive into a complex system and work out what bit is doing what. Systems are more loosely coupled than programs, so I tend to think that this is a somewhat different set of skills.
Troubleshooters need to be able to learn fast. Part of that is being able to research things, to figure out what articles or books or chapters have the stuff that you need to know right away, and what bits you can skim or omit.
It's certainly helped me to know a number of different computer languages and be reasonably familiar with a number of different systems. Pick nicely divergent ones, so that you get exposed to a bunch of different ideas.
(I have probably omitted a number of things. I may update this later, and comments are welcome.)
How many places actually send us email?
A few weeks ago I discovered that only 220 different IP addresses sent us actual email over the course of a week. This naturally raises the question: was this just a slow week, or is this typical? The answer turns out to be 'maybe'.
On the system I usually run my stats on, I only have logs going back about 28 days; looking at the entire time period, there was email from 443 different IP addresses. Not surprisingly, the distribution of how much email comes from where is very uneven, with almost all of the email we get is from a few mailing list hosts and the campus-wide email system.
On another system I have logs going back almost a year. Over that time, we got email from only 1,427 different IP addresses (only 95,000 email messages, though). On this system, the big source of email turns out to be Yahoo's webmail, and again things have a very sharp dropoff.
While this has practical uses for our specific situation, the more I think about it the less I think it really generalizes very well. Most of the people here use the central campus-wide email system and at most have their email forwarded from there to our systems; only a relatively few are still using our systems as their primary email system.
The usual quick rejection stats for 2005-07-16