Automation changes as systems grow

Over the time I've worked in system administration, I've observed a common pattern that I will summarize this way:

As your system count grows, you wind up automating the automation.

Here is the levels that I've seen people (myself included) go through:

• managing systems by hand, one by one.
• using tools that let you run a command on all of your systems or classes of your systems.
• automating things for clusters, so that you don't have to explicitly run commands on each system.

There are well-managed places that climb up these levels early, but in many cases it happens because you have no choice; you have enough systems that you can't afford to manage them with any lower level.

(I haven't personally worked in environments that went beyond the third level, but I imagine that when you grow bigger you can't afford to create clusters by hand so you wind up automating the cluster formation and configuration.)

What made this strikingly visible to me was moving backwards in the levels, when I went from an environment where there were enough systems that I was at the third level to an environment where there were few enough systems that they're mostly being managed at the first and second level. It's been somewhat of a culture shock.