== One reason why '_zpool status_' can hang

The '_zpool status_' command is [[infamous ZFSZpoolStalls]] for stalling
and hanging exactly when you need it the most, namely when something is
going wrong with your system. I've recently run down one reason why it
does this.

The culprit is my old friend [[ZFS GUIDs ZFSGuids]]. Information
about disks involved in ZFS pools includes both their GUIDs and their
theoretical device paths.  When '_zpool status_' prints out the user
friendly shortened device names, it doesn't just take the theoretical
path and trim most things off to get the device name; as I've [[alluded
to before ZFSGuidsVsNames]], it decides to be hyper-correct and check
that the device named in the configuration really is the right device.

In theory this is simple, as Solaris has some system calls for doing
pretty much all of the work. In practice these system calls require
you to open the disk device that you want to check, and under some
circumstances this _open()_ will stall for significant amounts of time
(several minutes, for example). An iSCSI target that isn't responding is
one such circumstance.

If you've ever seen this happen, you might wonder why '_zpool status_'
hangs completely immediately before printing any pool configuration
information, instead of getting to the point where it starts to print
device names for affected devices. The answer is that '_zpool status_'
is helpfully extra-clever; before it prints out any pool configuration
stuff, it pauses to work out how wide it has to make the device name
column so that everything will line up nicely. This requires working
out the friendly name of all devices, which requires that hyper-correct
checking of the configuration, which stalls the entire process if *any*
disk is very slow to _open()_.

(For extra fun, this 'calculate the needed width' step also looks at the
spare disks (if any), so a single bad spare disk, one that's not even in
use, can cause '_zpool status_' to stall on you.)

The '_zpool iostat_' command also does the same extra-clever step of
working out the maximum width of the device name column, so it will
stall for the same reason. For bonus points, '_zpool iostat_' does
this *every time it prints out a round of statistics*. Yes, really.
No wonder plain _iostat_ is acres better if anything bad is going
on.

By the way, this particular stall only happens if you have permissions
to open the device in the first place, ie it only happens if you are
root. So if you suspect ZFS problems, especially if you want '_zpool
iostat_' results, ~~run the commands as a non-root user~~.

(This is not the only way that _zpool status_ can stall; I've seen it
stutter when it was trying to get the ZFS pool configuration from the
kernel.)

=== Sidebar: where this is in the code

The _zpool_ source code is usr/src/cmd/zpool, and the whole width
calculation stuff is in [[zpool_main.c:max_width()|]]. This calls
[[zpool_vdev_name()|]], in [[lib/libzfs/common/libzfs_pool.c|]], which
calls [[path_to_devid()|]], which actually _open()_'s the device. This
check is thoughtfully guarded to make sure that it doesn't open devices
that ZFS has gotten around to declaring are actually bad; sadly, ZFS
makes such declarations long after _open()_'s of iSCSI target disks have
started stalling for minutes at a time.