Upgrading machines versus reinstalling them
Yesterday I mentioned that we would be 'upgrading' the version of OmniOS on our fileservers not by using the OmniOS upgrade process but by reinstalling them. While this was partly forced by an OmniOS problem, it's actually our approach in general. We tend to take this for two reasons.
The first reason is that it leads to either simpler install instructions or more identical machines if you have to rebuild one, depending on how you approach rebuilding upgraded machines. If you upgraded a machine from OS version A to OS version B, in theory you should reinstall a replacement by going through the same process instead of directly installing OS version B. If you directly install OS version B, you have a simpler and faster install process but you almost never get an exactly identical machine.
(In fact until you actually do this as a test you can't be sure you even wind up with a fully functional replacement machine. It's always possible that there's something vital in your current build instructions that only gets set up right if you start from OS version A and then upgrade.)
The second reason is that customizations done on OS version A are not always still applicable or necessary on OS version B. Sometimes they've even become counterproductive. If you're upgrading, you have to figure out how to find these issues and then how to fix them up. If you're directly (re)installing OS version B, you get a chance to start from scratch and apply only what you need (in the form you now need it in) on OS version B, and you don't have to deal with juggling all sorts of things during the transition from version A to version B.
(Relatedly, you may have changed your mind or simply learned better since your install of OS version A. Doing a from-scratch reinstall is a great opportunity to update to what you feel is the current best practice for something.)
Mind you, there are machines and situations where in-place upgrades are less disruptive and easier to do than complete reinstalls. One of them is when the machine has complex local state that is hard to fence off or back up and restore; another is if a machine was heavily customized, especially in ad-hoc ways. And in-place upgrades can involve less downtime (especially if you don't have surplus machines to do complex juggling). This is a lot of why I do in-place live upgrades of my workstations.