Our pragmatic approach to updating machines to match our baseline
A commentator on my entry on our approach to configuration management asked a good question:
the one thing that is problematic, is development of the "gold instalation standatd". when i make some changes, sometimes it's more work to get all the older machines to the new standard state. Do you solve this some way, or the machines are singletons even in the time?
Our answer is that we're pragmatic about this and as a result it depends on why we're changing the baseline installation. First off, changes to the baseline are basically always because of changes to at least some of the actual systems; the real question is thus not whether we update some systems to the new baseline but whether we update all of them to it. The answer to that depends on the change.
Some changes are things that we actively want on all of our systems (or all of the applicable type of system, like login servers) because they're driven by the users requesting things like 'can you add package X to the login servers' or us discovering we need to turn off some new vendor security feature. Obviously these get updated on all of the relevant servers (or at least all of the ones that we care strongly about); the update to the baseline is just to make sure any new or rebuilt servers also get this change. Some changes only really apply to certain sorts of machines but we updated the baseline to do them on every machine because it's easier that way and it does no harm. In this case we don't run around updating the machines the change doesn't really apply to, even though this means that a newly (re)built version of the machine will be different from the current version.
(In theory this is okay because the difference won't create any functional difference.)
One way of summarizing this is that we usually don't bother changing machines if we think that the change won't have any observable effect (in practice, not in theory; if we'll never notice whether or not a package is installed on a machine it qualifies, for example).