PCIe bus addresses,
lspci, and working out your PCIe bus topology
All PCIe devices have a PCIe bus address, which is shown by
dmidecode (cf), and so on.
As covered in the
lspci manpage, the fully
general form of PCIe bus addresses is
<domain>:<bus>:<device>.<function>. On most systems, the
domain is always 0000 and is omitted by
lpsci and what you see
is the bus, the device, and the function, which looks like this:
0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] [...]
That is the Radeon card on my office machine (or at least the video display portion of it), and it's function 0 of device 00 on bus 0a. Generally the device number and the function number are stable, but the bus number can definitely change depending on what other hardware you have in your machine. As I understand the situation, modern machines have many separate PCIe busses (behind PCIe bridges and other things), and a PCIe address's bus number depends on both the order that things are scanned by the BIOS and how many other busses and sub-busses there are on your system. Some cards have PCIe bridges and other things, and so whether or not you have one of them in your system (and where) can change how other bus numbers are assigned.
As covered in yesterday's entry on looking into PCIe slot topology,
lspci will print the actual topology of
your PCIe devices with '
lspci -tv'. Ever since I found out about
this, I've wondered how to go from the topology to the PCIe addresses
in plain '
lspci -v' output, and how I might verify the topology
or decode it from '
lspci -vv' output. As it turns out, both are
possible (and Matt's comment on yesterday's entry
gave me a useful piece of information).
So let's start from '
lpsci -tv' output, because it's more complex.
The first line of '
lspci -tv' looks like this:
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
The portion in brackets is the domain and bus that everything under this point in the tree is on. This is domain 0000 and bus 00, which is generally the root of the PCIe topology. Going along, we get to my first NVMe drive:
+-01.1-----00.0 Kingston Technology Company, Inc. Device 2263
This is not directly on bus 00; instead it is accessed through a device
at 01.1 on this bus, which thus has the (abbreviated) PCIe address of
lspci -v' tells me that this is a PCIe bridge, as expected:
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [...]
Much like the
s of the root of the tree, the '
' bit after
it in '
lspci -tv' means that all PCIe devices under this bridge are
on bus 01, and there is only one of them, the NVMe drive, which will
thus have the PCIe address 01:00.0:
01:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 2263 [...]
The X370 chipset controller presents a more complex picture:
+-01.3-[02-09]--+-00.0 Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller
The actual bridge is at 00:01.3 (another PCIe GPP bridge), but it has multiple busses behind it, from 02 through 09. If you look at the PCIe topology in yesterday's entry, you can predict that there are three things directly on bus 02, which are actually all functions of a single device; 02:00.0 is a xHCI controller, 02:00.1 is a SATA controller, and 02:00.2 is a 'X370 Series Chipset PCIe Upstream Port'. Behind it are a series of 'Chipset PCIe Port' devices (all on bus 03), and behind them are the actual physical PCIe slots and some onboard devices (a USB 3.1 host controller and the Intel gigabit Ethernet port). Each of these gets their own PCIe bus, 04 through 09, so for example my onboard Ethernet is 08:00.0 (bus 08, device 00, function 0):
08:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
Now let's go the other way, from '
lspci -vv' output to what device
is under what. As I showed above, my Radeon card is 0a:00.0. Its
upstream device is a PCIe GPP bridge at 00:03.1. If we examine that
GPP bridge in '
lspci -vv', we will see a 'Bus:' line:
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge (prog-if 00 [Normal decode]) [...] Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0 [...]
The primary bus is the 00: portion of its PCIe address, and the secondary bus is its direct downstream bus. So we would expect to find anything directly under this on bus 0a, which is indeed where the Radeon is.
A more interesting case is the 00:01.3 PCIe bridge to the X370 chipset. This reports:
Bus: primary=00, secondary=02, subordinate=09, sec-latency=0
What this appears to mean is that while this bridge's direct downstream bus is 02, somewhere below it are PCIe busses up to 09. I suspect that the PCIe specification requires that busses be assigned sequentially this way in order to make routing simpler.
If you have a device, such as my Radeon card at 0a:00.0, there is
no way I can see in the device's verbose PCIe information to find
what its parent is (end devices don't have a 'Bus:' line). You have
to search through the '
lspci -vv' of other devices for something
with a 'secondary=' of bus 0a. I think you'll pretty much always
find this somewhere, since generally something has to have this as
a direct downstream PCIe bus even if it's under a fan-out PCIe
setup like the X370 chipset bridge on this machine.
(Working backward this way instead of using '
lspci -tv' can be a
useful way of reassuring yourself that you really do understand the
topology. You may also want to look at the details of an upstream
device to find out, for example, why your Radeon card appears to
be running at PCIe 1.0 speeds. I haven't solved that mystery yet,
partly because I've been too busy to reboot my office machine to
get it into the BIOS.)