Some rough things about the naming of SAS drives on Linux

February 19, 2014

This isn't a comprehensive look at the names of SAS drives because SAS (as a technology) has a lot of options. Instead this is a look at how directly attached SATA drives behind SAS controllers get named. I assume that real SAS drives get named the same (but I don't have any to test with) and I have no idea how things look for SAS (or SATA) drives behind a SAS expander or more complex SAS topologies. Also some of this depends on the SAS controller and its driver; we're using LSI cards driven by the mpt2sas driver.

To start with, SAS drives have sdX names and kernel 'SCSI host names' just like SATA drives do. As with SATA drives, sdX drive names are not necessarily stable across hotswaps. Unlike SATA drives, the SCSI host names are not necessarily fully stable either; I've seen them get renumbered within a SCSI host after a hotswap, so that what was 'scsi6:0:3:0' becomes 'scsi6:0:8:0'. So far, LSI cards have a single SCSI host regardless of how many SAS ports they support and how those ports are physically connected (eg with a 4x SFF-8087 multiplexer or broken out as individual SAS ports).

(For background, see my basic overview of SAS and the names of Linux SATA devices, which covers kernel SCSI host names in more detail along with the three other device namespaces.)

All of this tells you nothing about the drive's physical slot. In SAS (or at least in mpt2sas for directly attached devices), physical location information is the SAS PHY number. To find this you must go grubbing around in sysfs, so I will just show you:

set -P                   # in bash
cd /sys/block/sdX/device
cd ../../..
ls phy-*

(There is probably a simpler path from the block device, but sysfs is the kind of thing where I find a working way and then stop.)

The PHY name will be of the form 'phy-6:0', meaning PHY 0 on SCSI host 6. PHY numbers sometimes also show up in kernel messages, such as:

scsi 6:0:2:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x50014ee25df6de01)
port-6:7: remove: sas_addr(0x4433221103000000), phy(3)

Mapping PHY numbers to actual physical hardware slots is something that you'll have to do yourself for your specific hardware. Please don't assume that PHY numbering matches how the card BIOS does it (as seen in BIOS information printouts during boot or if you go into the BIOS itself); for our LSI cards, it does not.

(Although it may be obvious, PHY numbers are reused between different SAS controllers. Several controllers may all have a PHY 0.)

Since the SCSI host name of a SAS drive in a given physical slot is not stable, /dev/disk/by-path sensibly does not use it for SAS drives. Instead it uses the 'SAS address' of each disk in combination with the PCI device number. The SAS address for each drive is exposed in sysfs as /sys/block/sdX/device/sas_address and on our hardware with mpt2sas appears to vary only due to the PHY number. You can see SAS addresses in the earlier kernel messages I gave; the first message results in a by-path filename that looks like 'pci-0000:07:00.0-sas-0x4433221101000000-lun-0' (for the whole disk, and the '01' portion appears to mark the PHY number).

Note that SAS addresses are only unique on their particular SAS controller. The system this message comes from has two SAS controllers and both controllers have disks with the SAS address of 0x4433221101000000.

(Possible interesting reading is this writeup based on what the mptsas driver does. What I've checked seems to match fairly well with what mpt2sas does on our system.)

Written on 19 February 2014.
« The reasoning problem with describing things with a programming language
You should segregate different traffic to different mailing lists »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Feb 19 23:58:51 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.