A while back I wrote about FutureSysadminJobs and suggested that people who wanted satisfying careers as system administrators over the long term should develop the skills to be troubleshooters. Which begs the question: what are those skills?

The first and most important skill is: you have to find all this interesting. Enjoying being a curious packrat is not entirely required, but I think it helps a whole lot, especially as you'll probably need to learn a number of things on your own time.

Other than that, based on my own experiences troubleshooting various issues I'm going to say:

Troubleshooters have to know how to program. Really program, not just write little scripts. Sometimes you'll write programs and sometimes you'll have to understand them, and you can't do either if you can't program yourself. (This is probably an unpopular view, but I feel that anyone who can't program is fundamentally crippled here.)

Troubleshooters have to know how to debug, which is harder than it looks. Debugging is part instincts and part paranoia and part obsessive completeness and almost entirely without useful textbooks, which means you have to learn it the hard way, by doing it.

Troubleshooters have to know how things work, because if you don't understand how things work you can't see where they can go wrong. (This means that you are going to be storing away a lot of trivia in your mind. It will help a lot if you like doing this.)

Troubleshooters need to know how to dive into big programs, zoom right in on the one little relevant bit, understand it, and then change it. This is a distinctly different skill than normal program maintenance, and like debugging you mostly get to develop it by being thrown in the deep end.

Similarly, you need to be able to dive into a complex system and work out what bit is doing what. Systems are more loosely coupled than programs, so I tend to think that this is a somewhat different set of skills.

Troubleshooters need to be able to learn fast. Part of that is being able to research things, to figure out what articles or books or chapters have the stuff that you need to know right away, and what bits you can skim or omit.

It's certainly helped me to know a number of different computer languages and be reasonably familiar with a number of different systems. Pick nicely divergent ones, so that you get exposed to a bunch of different ideas.

(I have probably omitted a number of things. I may update this later, and comments are welcome.)