It's my belief that every sysadmin who has a middling number of machines (more than one or two and less than a large fleet) sooner or later winds up with a set of tools to let them run commands on each of those machines or on subsets of those machines. I am no exception, and over the course of my career I have adopted, used, and built several iterations of this.

(People with large fleets of machines generally never run commands on them by hand but have some sort of automated fleet management system based around Puppet, Chef, CFEngine, Salt, Ansible, or the like. Sometimes these systems come with the ability to do this for you.)

My current iteration is based around simple shell scripts. The starting point is a shell script called machines; it prints out the names of machines that fall into various categories. The categorization of machines is entirely hand maintained, which has both problems and advantages. As a result the whole thing looks like:

mach() {
  for i in "$@" do
    case "$i" in
    apps) echo apps0 apps1 apps2 apps3 testapps;;
    ....
    ubuntu) echo `mach apps comps ...` ...;;
    ....
    *) echo $i;;
    esac
  done
}

mach "$@" | tr '\012' ' '; echo

(I put all of the work in a shell function so that I could call it recursively, for classes that are defined partly in terms of other classes. The 'ubuntu' class here is an example of that.)

So far we have few enough machines and few enough categories of machines that I'm interested in that this approach has not become unwieldy.

(There is also a script called mminus for doing subtractive set operations, so I can express 'all X machines except Y machines' or the like. This comes in handy periodically.)

The main script for actually doing things is called oneach, which does what you might think: given a list of machines it runs a command line on each of them via ssh. You can ask it to run the command in a pseudo-tty and without any special output handling, but normally it runs the command just with 'ssh machine command' and it prefixes all output with the name of the machine; you can see an example of that in this awk-based reformatting problem (an oneach run produced the input for my problem). Because I like neat formatting, oneach has an option to align the starting column of all output and I usually use that option (via a cover script called onea, because I'm lazy). The oneach script doesn't try to do anything fancy with concurrent execution or the like, it just does one ssh after the other.

Finally, I've found it useful to have another script that I call replicate. Replicate uses rsync to copy one or more files to destination machines or machine classes (it can also use scp for some obscure cases). replicate is handy for little things like pushing changes to dotfiles or scripts out to all of the machines where I have copies of them.

As a side note, machines has become a part of my dmenu environment. I use its list of machines as one of the the inputs to dmenu's autocompletion (both for normal logins and for special '@<machine>' logins as root), which makes it really quick and convenient to log into most of our machines (this large list of machines is part of the things I hide from dmenu's initial display of completion in a little UI tweak that turned out to be quite important for me).

Note that I don't necessarily suggest that you adopt my approach for running commands on your machines, which is one reason I'm not currently planning to put these scripts up in public. There are a lot of ways to solve this particular problem, many of them better and more scalable than what I have. I just think that you should get something better than manual for loops (which is what I was doing before I gave in and wrote machines, oneach, and so on).