Back in early 2021 I wrote about my impressions of NVMe versus SATA (or SAS) SSDs for basic servers. At that point I didn't expect us to get NVMe based servers any time soon, especially for servers not focused on fast storage. Well, times change, and we now have a number of 1U servers with U.2 NVMe drives. These aren't really "basic" servers in our usual sense; instead they tend to be pretty powerful compute servers. But they're still 1U servers and in theory there's nothing to stop people from having lower end ones with NVMe SSDs. Our experiences with these servers have been positive, in that everything works as we expect and basically how things would be if these were SATA SSDs instead.

(Obviously the U.2 NVMe drives are a lot faster and have lower latency, but these servers mostly don't put any real stress on their storage.)

We didn't get these servers with NVMe disks instead of SATA (or SAS) disks because we had some attraction to NVMe; if anything, we prefer SATA SSDs to U.2 NVMe SSDs because it's much easier to get spares and replacements (SATA SSDs are commodity items; U.2 NVMe SSDs are more expensive and harder to find). Instead, we got these servers with U.2 NVMe drives because that's the configuration they really wanted to come in. All of these servers have four hot swap drive bays (taking their own proprietary drive carriers), although we normally only use two (for a mirrored pair of system disks), and we opted to get them with four U.2 NVMe drives each in order to build up a pool of spares.

Physically and in operation these are just like conventional SATA or SAS drive carriers (from this particular system vendor) and more or less just like conventional 2.5" SATA and SAS drives (they may be thicker, but I don't pay close attention to that). In fact they're so physically similar that I'm glad the vendor puts a big 'NVMe' label on the front, because otherwise we could easily get confused about which drive carrier is U.2 NVMe and which drive carrier is SATA SSD.

One particular area which they are just like SATA drives in drive carriers is that we've hot-swapped inactive U.2 NVMe drives without problems. Linux certainly didn't explode. This gives me hope that we'll be able to deal gracefully if a system drive fails and has to be replaced. Hopefully, a failed NVMe drive won't have adverse consequences for the PCIe fabric it's connected to.

(Our hot-swapping of inactive drives came about because we left all four drives inserted in some servers, although we were only using two, and then later wanted to pull the two inactive drives out.)

I don't know why this particular vendor decided to make these systems be basically native U.2, although they're not really storage servers (being 1U systems with only four drive bays). All of the systems that are this way are dual-socket AMD Zen3 Epyc based ones, so maybe it's partly because they have so many PCIe lanes available.