Looking into your system's PCIe slot topology and PCIe lane count under Linux

December 5, 2019

Suppose, not hypothetically, that you want to understand the PCIe slot and bus topology of your Linux system and also work out how many PCIe lanes important things have been given. Sometimes the PCIe lane count doesn't matter, but things like NVMe drives may have lower latency under load and perform better if they get their full number of PCIe lanes (generally four lanes, 'x4'). You can do this under Linux, but it's not as straightforward as you'd like.

The easiest way to see PCIe topology is with 'lspci -tv', which shows you how PCIe slots relate to each other and also what's in them. On my office machine with the first configuration of PCIE to NVMe adapter card, this looked like the following:

-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
           +-00.2  [...]
           +-01.0  [...]
           +-01.1-[01]----00.0  Kingston Technology Company, Inc. Device 2263
           +-01.3-[02-09]--+-00.0  Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller
           |               +-00.1  Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller
           |               \-00.2-[03-09]--+-00.0-[04]----00.0  Kingston Technology Company, Inc. Device 2263
           |                               +-02.0-[05]--
           |                               +-03.0-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
           |                               +-04.0-[07]----00.0  ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller
           |                               +-06.0-[08]----00.0  Intel Corporation I211 Gigabit Network Connection
           |                               \-07.0-[09]--
           +-02.0  [...]
           +-03.0  [...]
           +-03.1-[0a]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
           +-04.0  [...]
[...]

(By the way, the PCIe numbers shown here have no resemblance to the numbers in the output of plain 'lspci'. Also, I have left out the text for a lot of 'PCIe Dummy Host Bridge' devices and a couple of others.)

The two Kingston PCIe devices are the two NVMe drives. The first one listed is in the motherboard M.2 slot; the second one listed is on its adapter card in a PCIe x16 @ x4 slot driven by the X370 chipset instead of directly by the CPU. Although it's not obvious from the topology listing, one of the Intel Ethernets is on the motherboard (but driven through the X370 chipset) and the other is a card in a PCIe x1 slot.

Now here's the same machine with the PCIe adapter for the second NVMe drive moved to a PCIe x16 @ x8 GPU card slot:

-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
           +-00.2  [...]
           +-01.0  [...]
           +-01.1-[01]----00.0  Kingston Technology Company, Inc. Device 2263
           +-01.3-[02-09]--+-00.0  Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller
           |               +-00.1  Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller
           |               \-00.2-[03-09]--+-00.0-[04]--
           |                               +-02.0-[05]--
           |                               +-03.0-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet Controller (Copper)
           |                               +-04.0-[07]----00.0  ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller
           |                               +-06.0-[08]----00.0  Intel Corporation I211 Gigabit Network Connection
           |                               \-07.0-[09]--
           +-02.0  [...]
           +-03.0  [...]
           +-03.1-[0a]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
           +-03.2-[0b]----00.0  Kingston Technology Company, Inc. Device 2263
           +-04.0  [...]
[...]

It's pretty clear that the second NVMe drive has shifted its position in the topology fairly significantly, and given that we have '03.1' and '03.2' labels for the Radeon GPU and the NVMe drive, it's not hard to guess that it's now in one of a pair of GPU slots. It's also clearly no longer under the X370 chipset, but instead is now on the same (PCIe) level as the first NVMe drive.

Working out how many PCIe lanes a card wants and how many it's actually getting is harder and more annoying. As far as I know, the best way of finding it out is to look carefully through the output of 'lspci -vv' for the device you're interested in and focus on the LnkCap and LnkSta portions, which will list what the card is capable of and what it actually got. For example, for my PCIe to NVMe adapter card in its first location (the first topology), where it was choked down from PCIe x4 to x2, this looks like the following:

04:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 2263 (rev 03) (prog-if 02 [NVM Express])
 [...]
   LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
           ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
 [...]
   LnkSta: Speed 5GT/s (downgraded), Width x2 (downgraded)
           TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 [...]

This says that it could do PCIe x4 at 8GT/s (which is PCIe 3.0) but was downgraded to PCIe x2 at 5GT/s (which is PCIe 2.0). In the PCIe adapter's new location in a GPU card slot where the NVMe drive could get full speed, the LnkCap didn't change but the LnkSta changed to:

   LnkSta: Speed 8GT/s (ok), Width x4 (ok)
           TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

So now the NVMe drive is getting x4 PCIE 3.0.

In general, searching 'lspci -vv' output for 'downgraded' can be interesting. For example, on my office machine, the Radeon GPU reports the following in both PCIe topologies:

0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] (rev c7) (prog-if 00 [VGA controller])
 [...]
   LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <1us
           ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
 [...]
   LnkSta: Speed 2.5GT/s (downgraded), Width x8 (ok)
           TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 [...]

I have no idea why the Radeon GPU has apparently been downgraded from PCIe 3.0 all the way to what looks like PCIe 1.0, although it may be because it's probably behind a 'PCIe GPP Bridge' that has also been downgraded to 2.5 GT/s. Perhaps I've missed some BIOS setting that is affecting things (BIOS settings can apparently influence this part of PCIe, as they do many bits of it). For my relatively basic X usage, this downgrade may not matter, but having noticed it I'm now curious and somewhat irritated. If my card can do PCIe 3.0, I want to be getting PCIe 3.0 if at all possible.

(Additional reading on this PCIe stuff under Linux includes here, here, and here. As usual, I'm writing this down so that I have it for reference the next time I need to poke around in this area.)

Written on 05 December 2019.
« Desktop motherboards can have fewer useful PCIe slots than they seem to
PCIe bus addresses, lspci, and working out your PCIe bus topology »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Dec 5 00:21:22 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.