Common motherboards are supporting more and more M.2 NVMe drive slots

December 6, 2024

Back at the start of 2020, I wondered if common (x86 desktop) motherboards would ever have very many M.2 NVMe drive slots, where by 'very many' I meant four or so, which even back then was a common number of SATA ports for desktop motherboards to provide. At the time I thought the answer was probably no. As I recently discovered from investigating a related issue, I was wrong, and it's now fairly straightforward to find x86 desktop motherboards that have as many as four M.2 NVMe slots (although not all four may be able to run at x4 PCIe lanes, especially if you have things like a GPU).

For example, right now it's relatively easy to find a page full of AMD AM5-based motherboards that have four M.2 NVMe slots. Most of these seem to be based on the high end X series AMD chipsets (such as the X670 or the X870, but I found a few that were based on the B650 chipset. On the Intel side, should you still be interested in an Intel CPU in your desktop at this point, there's also a number of them based primarily on the Z790 chipset (and some the older Z690). There's even a B760 based motherboard with four M.2 NVMe slots (although two of them are only x1 lanes and PCIe 3.0), and an H770 based one that manages to (theoretically) support all four M.2 slots at x4 lanes.

One of the things that I think has happened on the way to this large supply of M.2 slots is that these desktop motherboards have dropped most of their PCIe slots. These days, you seem to commonly get three slots in total on the kind of motherboard that has four M.2 slots. There's always one x16 slot, often two, and sometimes three (although that's physical x16; don't count on getting all 16 PCIe lanes in every slot). It's not uncommon to see the third PCIe slot be physically x4, or a little x1 slot tucked away at the bottom of the motherboard. It also isn't necessarily the case that lower end desktops have more PCIe slots to go with their fewer M.2 slots; they too seem to have mostly gone with two or three PCIe slots, generally with limited number of lanes even if they're physically x16.

(I appreciate having physical x16 slots even if they're only PCIe x1, because that means you can use any card that doesn't require PCIe bifurcation and it should work, although slowly.)

As noted by commentators on my entry on PCIe bifurcation and its uses for NVMe drives, a certain amount of what we used to need PCIe slots for can now be provided through high speed USB-C and similar things. And of course there are only so many PCIe lanes to go around from the CPU and the chipset, so those USB-C ports and other high-speed motherboard devices consume a certain amount of them; the more onboard devices the motherboard has the fewer PCIe lanes there are left for PCIe slots, whether or not you have any use for those onboard devices and connectors.

(Having four M.2 NVMe slots is useful for me because I use my drives in mirrored pairs, so four M.2 slots means I can run my full old pair in parallel with a full new pair, either in a four way mirror or doing some form of migration from one mirrored pair to the other. Three slots is okay, since that lets me add a new drive to a mirrored pair for gradual migration to a new pair of drives.)


Comments on this page:

By jmassey at 2024-12-07 11:57:47:

One of the things that I think has happened on the way to this large supply of M.2 slots is that these desktop motherboards have dropped most of their PCIe slots.

Sure, it happened along the way, but I don't think they're related. As you note, some boards have few PCIe slots and few M.2 slots. It's something I noticed while looking at AM4 boards. I and you have also both noted that it's not an inherent limitation of lanes, because trade-offs are possible—like SATA ports being disabled when lanes are needed elsewhere.

I suspect it's just that people don't need slots like they used to. I feel like my desire for PCIe slots is more a reflection of my history than a concrete need, because I remember the days when all boards would have 6 or 7 and we'd use a lot of them. A network card, a modem, a sound card, a CD-ROM controller, a video card, maybe a Hercules card if someone wanted a second monitor. Especially when the slots were of 2 different types, one could easily run out. (Making them all work at the same time was another story, as was keeping a useful amount of RAM available.)

Now, every desktop board has one ethernet port (sometimes two, and often wi-fi), ample SATA ports, built-in sound, sometimes even video. Realistically, I think most manufacturers are targeting two graphics cards and a spare slot, which most people won't fill up. Network cards are probably the main other type people are still using—if they need a second port, or SFP, or anything faster than 2.5Gbe (I don't think people do "serious" networking over USB yet, but it may happen).

Having 4 M.2 slots around, that you might only use for a few days while re-doing your RAID, seems like a bit of a "luxury". Nice if you can get it, but probably not something that drives many motherboard purchasing decisions; you could find some other method if you had to. Out of curiosity, do you and your colleagues have any interesting PCIe requirements that haven't already been mentioned?

By cks at 2024-12-07 14:12:05:

At work, so far our server needs are simple enough so far that we only use one or two PCIe cards in current generation hardware. Our SLURM GPU nodes have an x16 GPU and a 10G card (which is usually x4), while firewalls tend to have a dual 10G card (at least x4). In the past we've had machines with a dual 10G card and a SAS controller card, but these days our fileservers have all of their SAS and SATA ports on the motherboard, and the 10G too.

Some of our backup-related servers may be running into SATA write speed limits, so it's possible that someday we'll want to put some NVMe drives into them. Since I doubt they do PCIe bifurcation, that'd probably be two x4 PCIe cards (although for this usage we might use only one NVMe drive), plus a 10G card.

Written on 06 December 2024.
« Buffered IO in Unix before V7 introduced stdio
PCIe cards we use and have used in our servers »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Fri Dec 6 23:27:41 2024
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.