2019-12-12
Linux makes your PCIe topology visible in sysfs (/sys
)
Getting some NVMe drives for my office machine
has been an ongoing education into many areas of PCIe, including
how to see your PCIe topology with lspci
and
understanding how PCIe bus addresses and topology relate to each
other. Today I coincidentally discovered
that there is another way to look into your system's PCIe topology,
because it turns out that the Linux kernel materializes it a directory
hierarchy in the sysfs filesystem that is usually mounted on /sys
.
Generally, the root of the PCI(e) bus hierarchy is going to be found
at /sys/devices/pci0000:00
. Given an understanding of PCIe
addresses, we can see that 0000:00 is the
usual domain and starting PCIe bus number. In this directory are a
whole bunch of subdirectories, named after the full PCIe bus address
of each device, so you get directories named things like '0000:00:03.2
'.
If you take of the leading '0000:', this corresponds to what 'lspci
-v
' will report as device 00:03.2. For PCIe devices that act as
bridges, there will be subdirectories for the PCIe devices behind
the bridge, with the full PCIe address of those devices. So in my
office machine's current PCIe topology, there
is a '0000:0b:00.0
' subdirectory in the 0000:00:03.2
directory,
which is my second NVMe drive behind the 00:03.2 PCIe bridge.
(And behind 0000:00:03.1
is my Radeon graphics card, which actually
has two exposed PCIe functions; 0000:0a:00.0
is the video side,
while 0000:0a:00.1
is 'HDMI/DP Audio'.)
There are a number of ways to use this /sys
information, some of
which are for future entries. The most obvious use is to confirm
your understanding of the topology and implied PCIe bus addresses
that 'lspci -tv
' reports. If the /sys
directory hierarchy matches
your understanding of the output, you have it right. If it doesn't,
something is going on.
The other use is a brute force way of finding out what the topology
of a particular final PCIe device is, by simply finding it in the
hierarchy with 'find /sys/devices/pci0000:00 -name ..
', where the
name is its full bus address (with the 0000: on the front). So, for
example, if we know we have an Ethernet device at 06:00.0, we can
find where it is in the topology with:
; cd /sys/devices/pci0000:00 ; find . -type d -name 0000:06:00.0 -print ./0000:00:01.3/0000:02:00.2/0000:03:03.0/0000:06:00.0
(Using '-type d
' avoids having to filter out some symlinks for
the PCIe node in various contexts; in this case it shows up as
'0000:00:00.2/iommu/ivhd0/devices/0000:06:00.0'.)
This shows us the path through the PCIe topology from the root, through 00:01.3, then 02:00.2, then finally 03:03.0. This complex path is because this is a device hanging off the AMD X370 chipset instead of off of the CPU, although not all chipset attached PCIe devices will have such a long topology.
Until I looked at the lspci
manpage more carefully, I was going
to say that this was the easiest way to go from a PCIe bus address
to the full path to the device with all of the PCIe bus addresses
involved. However, it turns out that in sufficiently modern versions
of lspci
, 'lspci -PP
' will report the same information in a
shorter and more readable way:
; lspci -PP -s 06:00.0 00:01.3/02:00.2/03:03.0/06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
Unfortunately the version of lspci
on our Ubuntu 18.04 machines
is not sufficiently modern; on those machines, a find
remains the
easiest way. You can do it from the output of either 'lspci -tv
'
or 'lspci -v
', as described in an earlier entry, but you have to do some manual work to
reconstruct all of the PCIe bus addresses involved.