Naming disk devices: drive IDs versus drive locations

October 31, 2013

From my perspective there are two defensible ways of naming disk drives at the operating system level. You can do it by a stable identifier tied to the physical drive somehow, such as a drive serial number or WWN, or by a stable identifier based on its connection topology and thus ultimately the drive's physical location (such as the 'port X on card Y' style of name). I don't want to get into an argument about which one is 'better' because I don't think that argument is meaningful; the real question to ask is which form of naming is more useful under what circumstances.

(Since the boundaries between the two sorts of names may be fuzzy, my rule of thumb is that it is clearly a drive identifier if you have to ask the drive for it. Well, provided that you are actually speaking to the drive instead of a layer in between. The ultimate drive identifiers are metadata that you've written to the drive.)

Before I get started, though, let me put one inconvenient fact front and center: in almost all environments today, you're ultimately going to be dealing with drives in terms of their physical location. For all the popularity of drive identifiers as a source of disk names (among OS developers and storage technologies), there are very few environments right now where you can tell your storage system 'pull the drive with WWN <X> and drop it into my hands' and have that happen. As I tweeted I really do need to know where a particular disk actually is.

This leads to my bias, which is that using drive identifiers makes the most sense when the connection topology either changes frequently or is completely opaque, or both. If your connection topology rearranges itself on a regular basis then it can't be a source of stable identifiers because it itself isn't stable. However you can sometimes get around this by finding a stable point in the topology; for example, iSCSI target names (and LUNs) are a stable point whereas the IP addresses or network interfaces involved may not be.

(Topology rearrangement can be physical rearrangement, ranging from changing cabling all the way up to physically transferring disks between enclosures for whatever reason.)

Conversely, physical location makes the most sense when topology is fixed (and drives aren't physically moved around). With stable locations and stable topology to map to locations, all of the important aspects of a drive's physical location can be exposed to you so you can see where it is, what the critical points are for connecting to it, what other drives will be affected if some of those points fail or become heavily loaded, and so on. Theoretically you don't have to put this in the device name if it's visible in some other way, but in practice visible names matter.

My feeling is that stable topology is much more common than variable topology, at least once you identify the useful fixed points in connection topology. Possibly this is an artifact of the environment I work in; on the other hand, I think that relatively small and simple environments like mine are much more common than large and complex ones.

Sidebar: the cynic's view of OS device naming

It's much easier to give disks an identifier based device name than it is to figure out how to decode a particular topology and then represent the important bits of it in a device name, especially if you're working in a limited device naming scheme (such as 'cXtYdZ', for example). And you can almost always find excuses for why the topology might be unstable in theory (eg 'the sysadmin might move PCI cards between slots and oh no').


Comments on this page:

Solaris 11 has a topology framework which among other things allows to identify a physical disk locations, it is even integrated with some tools.

For example x3-2l server with 26x internal SAS disks presented as JBOD (2x last disks are configured in RAID1 on HBA), to get mappings between WWNs and physical locations:

# diskinfo
D:devchassis-path            c:occupant-compdev
---------------------------  ---------------------
/dev/chassis/SYS/HDD00/disk  c0t5000CCA025D0CB40d0
/dev/chassis/SYS/HDD01/disk  c0t5000CCA025E000E8d0
/dev/chassis/SYS/HDD02/disk  c0t5000CCA025DCFAD4d0
/dev/chassis/SYS/HDD03/disk  c0t5000CCA025D1AD80d0
/dev/chassis/SYS/HDD04/disk  c0t5000CCA025DEC94Cd0
/dev/chassis/SYS/HDD05/disk  c0t5000CCA025DEC5A0d0
/dev/chassis/SYS/HDD06/disk  c0t5000CCA025DED97Cd0
/dev/chassis/SYS/HDD07/disk  c0t5000CCA025D1C300d0
/dev/chassis/SYS/HDD08/disk  c0t5000CCA025DED184d0
/dev/chassis/SYS/HDD09/disk  c0t5000CCA025BD1258d0
/dev/chassis/SYS/HDD10/disk  c0t5000CCA025DEDC68d0
/dev/chassis/SYS/HDD11/disk  c0t5000CCA025DEC674d0
/dev/chassis/SYS/HDD12/disk  c0t5000CCA025DEC004d0
/dev/chassis/SYS/HDD13/disk  c0t5000CCA025DEC808d0
/dev/chassis/SYS/HDD14/disk  c0t5000CCA025AB0548d0
/dev/chassis/SYS/HDD15/disk  c0t5000CCA025DECDACd0
/dev/chassis/SYS/HDD16/disk  c0t5000CCA03C02D0B4d0
/dev/chassis/SYS/HDD17/disk  c0t5000CCA025C94A98d0
/dev/chassis/SYS/HDD18/disk  c0t5000CCA03C02DABCd0
/dev/chassis/SYS/HDD19/disk  c0t5000CCA025D005F0d0
/dev/chassis/SYS/HDD20/disk  c0t5000CCA03C02C25Cd0
/dev/chassis/SYS/HDD21/disk  c0t5000CCA03C02661Cd0
/dev/chassis/SYS/HDD22/disk  c0t5000CCA025D590B8d0
/dev/chassis/SYS/HDD23/disk  c0t5000CCA03C02F614d0
/dev/chassis/SYS/HDD24       -
/dev/chassis/SYS/HDD25       -

Each HDDNN corresponds to a physical slot in the server. Now lets see ZFS pool status:

# zpool status pool-0
  pool: pool-0
state: ONLINE
  scan: scrub repaired 0 in 3h15m with 0 errors on Sat Oct 26 05:42:37 2013
config:

       NAME                       STATE     READ WRITE CKSUM
       pool                   -0  ONLINE       0     0     0
         mirror-0                 ONLINE       0     0     0
           c0t5000CCA025D0CB40d0  ONLINE       0     0     0
           c0t5000CCA025E000E8d0  ONLINE       0     0     0
         mirror-1                 ONLINE       0     0     0
           c0t5000CCA025DCFAD4d0  ONLINE       0     0     0
           c0t5000CCA025D1AD80d0  ONLINE       0     0     0
         mirror-2                 ONLINE       0     0     0
           c0t5000CCA025DEC94Cd0  ONLINE       0     0     0
           c0t5000CCA025DEC5A0d0  ONLINE       0     0     0
         mirror-3                 ONLINE       0     0     0
           c0t5000CCA025DED97Cd0  ONLINE       0     0     0
           c0t5000CCA025D1C300d0  ONLINE       0     0     0
         mirror-4                 ONLINE       0     0     0
           c0t5000CCA025DED184d0  ONLINE       0     0     0
           c0t5000CCA025BD1258d0  ONLINE       0     0     0
         mirror-5                 ONLINE       0     0     0
           c0t5000CCA025DEDC68d0  ONLINE       0     0     0
           c0t5000CCA025DEC674d0  ONLINE       0     0     0
         mirror-6                 ONLINE       0     0     0
           c0t5000CCA025DEC004d0  ONLINE       0     0     0
           c0t5000CCA025DEC808d0  ONLINE       0     0     0
         mirror-7                 ONLINE       0     0     0
           c0t5000CCA025AB0548d0  ONLINE       0     0     0
           c0t5000CCA025DECDACd0  ONLINE       0     0     0
         mirror-8                 ONLINE       0     0     0
           c0t5000CCA03C02D0B4d0  ONLINE       0     0     0
           c0t5000CCA025C94A98d0  ONLINE       0     0     0
         mirror-9                 ONLINE       0     0     0
           c0t5000CCA03C02DABCd0  ONLINE       0     0     0
           c0t5000CCA025D005F0d0  ONLINE       0     0     0
         mirror-10                ONLINE       0     0     0
           c0t5000CCA03C02C25Cd0  ONLINE       0     0     0
           c0t5000CCA03C02661Cd0  ONLINE       0     0     0
       spares
         c0t5000CCA025D590B8d0    AVAIL
         c0t5000CCA03C02F614d0    AVAIL

errors: No known data errors

There is -l option to zpool status which will display physical locations instead:

# zpool status -l pool-0
  pool: pool-0
state: ONLINE
  scan: scrub repaired 0 in 3h15m with 0 errors on Sat Oct 26 05:42:37 2013
config:

       NAME                             STATE     READ WRITE CKSUM
       pool-0                           ONLINE       0     0     0
         mirror-0                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD00/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD01/disk  ONLINE       0     0     0
         mirror-1                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD02/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD03/disk  ONLINE       0     0     0
         mirror-2                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD04/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD05/disk  ONLINE       0     0     0
         mirror-3                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD06/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD07/disk  ONLINE       0     0     0
         mirror-4                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD08/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD09/disk  ONLINE       0     0     0
         mirror-5                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD10/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD11/disk  ONLINE       0     0     0
         mirror-6                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD12/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD13/disk  ONLINE       0     0     0
         mirror-7                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD14/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD15/disk  ONLINE       0     0     0
         mirror-8                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD16/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD17/disk  ONLINE       0     0     0
         mirror-9                       ONLINE       0     0     0
           /dev/chassis/SYS/HDD18/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD19/disk  ONLINE       0     0     0
         mirror-10                      ONLINE       0     0     0
           /dev/chassis/SYS/HDD20/disk  ONLINE       0     0     0
           /dev/chassis/SYS/HDD21/disk  ONLINE       0     0     0
       spares
         /dev/chassis/SYS/HDD22/disk    AVAIL
         /dev/chassis/SYS/HDD23/disk    AVAIL

errors: No known data errors

You can query for more information like serial numbers, fw versions, etc. - see croinfo(1M) man page.

This works out of the box on Oracle servers (both x86 and SPARC), for 3rd party servers if disks are directly attached without an expander you can create an XML file called Bay Labels which will describe the mappings and then all the tooling will just work. If there is a SAS expander then it is a little bit more complicated.

See also http://dtrace.org/blogs/eschrock/2008/07/13/external-storage-enclosures-in-solaris/

And yes, I do agree that this is very helpful to have when building file servers, etc. as it allows to quickly and easily to identify a disk.

I'm not sure about your combo of storage chassis/drive enclosures, but the Dell, EMC mid-range, STK, Sun, etc, boxen I used to deal with had a way to flash the drive lights, or light a specific drive.

That meant I didn't have to worry about this sort of thing for the reason you're mentioning.

Written on 31 October 2013.
« An open question: part uniformity versus unit cost
Our likely future backend and fileserver hardware »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Oct 31 01:14:16 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.