Wandering Thoughts archives

2016-04-22

What Illumos/OmniOS PCI device names seem to mean

When working on an OmniOS system, under normal circumstances you'll use friendly device names from /dev and things like dladm (for network devices). However, Illumos-based systems have an underlying hardware based naming scheme (exposed in /devices), and under some circumstances you can wind up dealing with it. When you do, you'll be confronted with relatively opaque names like '/pci@0,0/pci8086,e04@2/pci8086,115e@0' and very little clue what these names actually mean, at least if you're not already an Illumos/Solaris expert.

So let's take just one bit here: pci8086,e04@2. The pci8086,e04 portion is the PCI subsystem vendor and device code, expressed in hex. You'll probably see '8086' a lot, because it's the vendor code for Intel. Then the @2 portion is PCI path information expressed relative to the parent. This can get complicated, because 'path relative to the parent' doesn't map well to the kinds of PCI names you get on Linux from eg 'lspci'. When you see a '@...' portion with a comma, that is what other systems would label as 'device.function'. If there is no comma in the '@..' portion, the function is implicitly 0.

(Note that the PCI subsystem vendor and device is different from the PCI vendor and device. Linux 'lspci -n' shows only the vendor code, because that's what's important for knowing what sort of thing it is instead of who exactly made it; you have to use 'lspci -vn' to see the subsystem stuff. Illumos's PCI names here are inherently framed as a PCI tree, whereas Linux lspci normally does not show the tree topology, just flat slot numbering. See 'lspci -t' for the tree view.)

As far as I can tell, in a modern PCI Express setup the physical slot you put a card into will determine the first two elements of the PCI path. '/pci@0,0' is just a (synthetic) PCI root instance, and then '/pci8086,e04@2' is a specific PCI Express Port. However, I'm not sure if one PCI Express Port can feed multiple slots and if it can, I'm not sure how you tell them apart. I'm not quite sure how things work for plain PCI cards, but for onboard PCI devices you get PCI paths like '/pci@0,0/pci15d9,714@1a' where the '@1a' corresponds to what Linux lspci sees as 00:1a.0.

So, suppose that you have a collection of OmniOS servers and you want to know if they have exactly the same PCI Express cards in exactly the same slots (or, say, exactly the same Intel 1G based network cards). If you look at /etc/path_to_inst and see exactly the same PCI paths, you've got what you want. If you look at the paths and see two systems with say:

s1: /pci@0,0/pci8086,e04@2/pci8086,135e@0
s2: /pci@0,0/pci8086,e04@2/pci8086,115e@0

What you have is a situation where the cards are in the same slots (because the first two elements of the path are the same) but they're slightly different generations and Intel has changed the PCI subsystem device code on you (seen in ',135e' versus ',115e'). If you're transplanting system disks from s2 to s1, this can cause problems that you'll need to deal with by editing path_to_inst.

I don't know what order Illumos uses when choosing how to assign instances (and thus eg network device names) to hardware when you have multiple instances of the same hardware. On a single card with multiple ports it seems consistent that the port with the lower function is assigned first, eg if you have a dual port card where the ports are pci8086,115e@0 and pci8086,115e@0,1, the @0 port will always be a lower instance than the @0,1 port. How multiple cards are handled is not clear to me and I can't reverse engineer it based on our current hardware.

(While we have multiple Intel 1G dual-port cards in our OmniOS fileservers, they are in PCI Express slots that differ both in the PCI subdevice and in the PCI path information; we have pci8086,e04@2 as the PCI Express Port for the first card and pci8086,e0a@3,2 for the second. I suspect that the PCI path information ('@2' versus '@3,2') determines things here, but I don't know for sure.)

PS: Yes, all of this is confusing (at least to me). Maybe I need to read up on general principles of PCI, PCI Express, and how all the topology stuff works (the PCI bus world is clearly not flat any more, if it ever was).

solaris/IllumosPCIDeviceNaming written at 02:02:19; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.