Wandering Thoughts archives

2019-12-12

Linux makes your PCIe topology visible in sysfs (/sys)

Getting some NVMe drives for my office machine has been an ongoing education into many areas of PCIe, including how to see your PCIe topology with lspci and understanding how PCIe bus addresses and topology relate to each other. Today I coincidentally discovered that there is another way to look into your system's PCIe topology, because it turns out that the Linux kernel materializes it a directory hierarchy in the sysfs filesystem that is usually mounted on /sys.

Generally, the root of the PCI(e) bus hierarchy is going to be found at /sys/devices/pci0000:00. Given an understanding of PCIe addresses, we can see that 0000:00 is the usual domain and starting PCIe bus number. In this directory are a whole bunch of subdirectories, named after the full PCIe bus address of each device, so you get directories named things like '0000:00:03.2'. If you take of the leading '0000:', this corresponds to what 'lspci -v' will report as device 00:03.2. For PCIe devices that act as bridges, there will be subdirectories for the PCIe devices behind the bridge, with the full PCIe address of those devices. So in my office machine's current PCIe topology, there is a '0000:0b:00.0' subdirectory in the 0000:00:03.2 directory, which is my second NVMe drive behind the 00:03.2 PCIe bridge.

(And behind 0000:00:03.1 is my Radeon graphics card, which actually has two exposed PCIe functions; 0000:0a:00.0 is the video side, while 0000:0a:00.1 is 'HDMI/DP Audio'.)

There are a number of ways to use this /sys information, some of which are for future entries. The most obvious use is to confirm your understanding of the topology and implied PCIe bus addresses that 'lspci -tv' reports. If the /sys directory hierarchy matches your understanding of the output, you have it right. If it doesn't, something is going on.

The other use is a brute force way of finding out what the topology of a particular final PCIe device is, by simply finding it in the hierarchy with 'find /sys/devices/pci0000:00 -name ..', where the name is its full bus address (with the 0000: on the front). So, for example, if we know we have an Ethernet device at 06:00.0, we can find where it is in the topology with:

; cd /sys/devices/pci0000:00
; find . -type d -name 0000:06:00.0 -print
./0000:00:01.3/0000:02:00.2/0000:03:03.0/0000:06:00.0

(Using '-type d' avoids having to filter out some symlinks for the PCIe node in various contexts; in this case it shows up as '0000:00:00.2/iommu/ivhd0/devices/0000:06:00.0'.)

This shows us the path through the PCIe topology from the root, through 00:01.3, then 02:00.2, then finally 03:03.0. This complex path is because this is a device hanging off the AMD X370 chipset instead of off of the CPU, although not all chipset attached PCIe devices will have such a long topology.

Until I looked at the lspci manpage more carefully, I was going to say that this was the easiest way to go from a PCIe bus address to the full path to the device with all of the PCIe bus addresses involved. However, it turns out that in sufficiently modern versions of lspci, 'lspci -PP' will report the same information in a shorter and more readable way:

; lspci -PP -s 06:00.0
00:01.3/02:00.2/03:03.0/06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

Unfortunately the version of lspci on our Ubuntu 18.04 machines is not sufficiently modern; on those machines, a find remains the easiest way. You can do it from the output of either 'lspci -tv' or 'lspci -v', as described in an earlier entry, but you have to do some manual work to reconstruct all of the PCIe bus addresses involved.

linux/PCIeTopologyInSysfs written at 00:22:48; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.