2019-03-29
Our current approach for significantly upgrading or modifying servers
Every so often we need to make some significant upgrade or change to one of our servers, for instance to upgrade from Ubuntu version to Ubuntu version. When we do this, we do two things. The first is that we reinstall from scratch rather than try to upgrade the machine's current OS and setup in place. There are a whole bunch of reasons for this (for any OS, not just Linux), including that it gets you as close as possible to insuring that the current state of the machine isn't dependent on its history.
(A machine that has been through major upgrades inevitably and invariably carries at least some traces of its past, traces that will not be there on a new version that was reinstalled from scratch.)
The second is that we almost always install the new instance of the server on new hardware and swap it into place, rather than reinstalling on the same hardware that is currently the live server. There are exceptions, usually for our generic compute servers, but for anything important we prefer new hardware (this is somewhat of a change from the past). One part of this is that using a new set of hardware makes it easy to refresh the hardware, change the RAM or SSD setup, and so on (and also to put the new server in a different place in your racks). Another part is that when you have two servers, rolling back an upgrade that turns out to have problems is much easier and faster than if you have destroyed the old server in the process of installing the new one. A third reason is more prosaic; there's always less downtime involved in a machine swap than in a reinstall from scratch, and among other things this leads to less or no pressure when you're installing the machine.
One consequence of our approach is that we always have a certain amount of 'not in production' replaced servers that are still in our racks but powered off and disconnected. We don't pull replaced servers immediately, in case we have to roll back to them, so after a while we have to remember that probably we should pull the old version of an upgraded server. We don't always, so every so often we basically wind up weeding our racks, pulling old servers that don't need to be there. One trigger for this weeding is when we need room in a specific rack and it happens to be cluttered up with obsolete servers. Another is when we run short on spare server hardware to turn into more new servers.
(Certain sorts of servers are recycled almost immediately in order to reclaim desirable bits of hardware in them. For example, right now anything with a 10G-T card is probably going to be pulled shortly after an upgrade in order to extract the card, because we don't have too many of them. There was a time when SSDs would have prompted recycling, but not any more.)
PS: We basically never throw out (still) working servers, even very old ones, but they do get less and less desirable over time and so sit deeper and deeper in the depths of our spare hardware storage. The current fate of really old servers is mostly to be loaned or passed on to other people here who need them and who don't mind getting decade old hardware (often with very little RAM by modern standards, which is another reason they get less desirable over time).
PPS: I'm not joking about decade old servers. We recently passed some Dell 1950s on to someone who needed scratch machines.
My NVMe versus SSD uncertainty (and hesitation)
One of the things I sometimes consider for my current home machine is adding a mirrored pair of NVMe drives (to go with my mirrored pair of SSDs and mirrored pair of HDs). So far I haven't really gotten very serious about this, though, and one of the reasons is that I'm not sure how much faster NVMe drives are going to be for me in practice and thus whether it is worth getting higher end NVMe drives or not.
(Another reason for not doing anything is that the cost per GB keeps dropping, so my inertia keeps being rewarded with more potential storage for the same potential cost.)
NVMe drives clearly have the possibility of being dramatically faster than SATA SSDs can be; their performance numbers generally cite large bandwidth, high number of IOPs, and a low latency (sometimes implicitly, sometimes explicitly). Assuming that these are relatively real, there are two obstacles to achieving them in practice; the OS driver stack and your (my) actual usage.
In order to delivery high bandwidth, many IOPs, and low latency for disk IO, the OS needs to have a highly efficient block IO system and good drivers. Extremely high-speed (and extremely parallel) 'disk' IO devices upend various traditional OS assumptions about how disk IO should be organized and optimized, and I'm not sure what the state of this is today for Linux. NVMe drives are now widely used, especially on high performance servers, so in theory things should be reasonably good here.
(Of course, one answer is that OS performance is only going to improve from here even with today's NVMe drives. But then my inertia whispers in my ear that I might as well wait for that to happen before I get some NVMe drives.)
The actual usage issue is that if you don't use your drives, how well they perform is mostly irrelevant. I have divided thoughts here. On the one hand, I don't really do much IO on my home machine these days, and much of what I do is not high priority (I may compile Firefox from source periodically, but I generally don't care if that's a bit faster or slower since I'm not actually waiting for it). On the other hand, I might well notice a significant improvement in latency even for infrequent disk IO activity, because it's activity that I'm waiting on (whether or not I realize it). Modern systems and programs go to disk for all sorts of reasons, often without making it clear to you that they are.
(In the days of loud HDs you could hear your disks working away, but SSDs are silent.)
Even given this, I do sometimes do IO intensive activities that I would really quite like to go fast. One periodic case of this is upgrading Fedora from release to release, which requires updating most of what is now over 5,000 RPM packages (with over 760,000 files and directories). This is a disk intensive process and I saw a real speed difference going from HDs to SSDs; I wouldn't mind seeing another one with a move to NVMe, assuming that NVMe would or could actually deliver one here.
PS: Since my major use of any new space will be for a ZFS pool, another question is how well ZFS itself is prepared for the performance characteristics of NVMe drives (well, ZFS on Linux). See, for example, this recent ZoL issue.