2016-04-22
What Illumos/OmniOS PCI device names seem to mean
When working on an OmniOS system, under normal circumstances you'll
use friendly device names from /dev
and things like dladm
(for
network devices). However, Illumos-based systems have an underlying
hardware based naming scheme (exposed in /devices
), and under
some circumstances you can wind up
dealing with it. When you do, you'll be confronted with relatively
opaque names like '/pci@0,0/pci8086,e04@2/pci8086,115e@0
' and
very little clue what these names actually mean, at least if you're
not already an Illumos/Solaris expert.
So let's take just one bit here: pci8086,e04@2
. The pci8086,e04
portion is the PCI subsystem vendor and device code, expressed in
hex. You'll probably see '8086' a lot, because it's the vendor code for
Intel. Then the @2 portion is PCI path information expressed relative
to the parent. This can get complicated, because 'path relative to the
parent' doesn't map well to the kinds of PCI names you get on Linux
from eg 'lspci
'. When you see a '@...' portion with a comma, that is
what other systems would label
as 'device.function'. If there is no comma in the '@..' portion, the
function is implicitly 0.
(Note that the PCI subsystem vendor and device is different from
the PCI vendor and device. Linux 'lspci -n
' shows only the vendor
code, because that's what's important for knowing what sort of thing
it is instead of who exactly made it; you have to use 'lspci -vn
'
to see the subsystem stuff. Illumos's PCI names here are inherently
framed as a PCI tree, whereas Linux lspci
normally does not show
the tree topology, just flat slot numbering. See 'lspci -t
' for
the tree view.)
As far as I can tell, in a modern PCI Express setup the physical
slot you put a card into will determine the first two elements of
the PCI path. '/pci@0,0' is just a (synthetic) PCI root instance,
and then '/pci8086,e04@2' is a specific PCI Express Port. However,
I'm not sure if one PCI Express Port can feed multiple slots and
if it can, I'm not sure how you tell them apart. I'm not quite sure
how things work for plain PCI cards, but for onboard PCI devices
you get PCI paths like '/pci@0,0/pci15d9,714@1a' where the '@1a'
corresponds to what Linux lspci
sees as 00:1a.0.
So, suppose that you have a collection of OmniOS servers and you want to know if they have exactly
the same PCI Express cards in exactly the same slots (or, say,
exactly the same Intel 1G based network cards). If you look at
/etc/path_to_inst
and see exactly the same PCI paths, you've
got what you want. If you look at the paths and see two systems
with say:
s1: /pci@0,0/pci8086,e04@2/pci8086,135e@0
s2: /pci@0,0/pci8086,e04@2/pci8086,115e@0
What you have is a situation where the cards are in the same slots
(because the first two elements of the path are the same) but they're
slightly different generations and Intel has changed the PCI subsystem
device code on you (seen in ',135e' versus ',115e'). If you're
transplanting system disks from s2 to s1, this can cause problems
that you'll need to deal with by editing path_to_inst
.
I don't know what order Illumos uses when choosing how to assign instances (and thus eg network device names) to hardware when you have multiple instances of the same hardware. On a single card with multiple ports it seems consistent that the port with the lower function is assigned first, eg if you have a dual port card where the ports are pci8086,115e@0 and pci8086,115e@0,1, the @0 port will always be a lower instance than the @0,1 port. How multiple cards are handled is not clear to me and I can't reverse engineer it based on our current hardware.
(While we have multiple Intel 1G dual-port cards in our OmniOS fileservers, they are in PCI Express slots that differ both in the PCI subdevice and in the PCI path information; we have pci8086,e04@2 as the PCI Express Port for the first card and pci8086,e0a@3,2 for the second. I suspect that the PCI path information ('@2' versus '@3,2') determines things here, but I don't know for sure.)
PS: Yes, all of this is confusing (at least to me). Maybe I need to read up on general principles of PCI, PCI Express, and how all the topology stuff works (the PCI bus world is clearly not flat any more, if it ever was).