I mentioned SATA port multipliers in passing yesterday but then I just assumed that everyone knew what they are (as I have done before). So, time for the explanations.

Put simply, SATA port multipliers let you talk to several disks over a single SATA port and cable. If you're old enough, you may now be asking why we'd want to get rid of one of SATA's advantages and go back to the bad old days of things like SCSI and multi-device IDE. The short answer is density; using port multipliers lets you talk to a lot more SATA disks than you could otherwise. Our iSCSI backends are a perfect illustration; with port multiplication it only takes us one PCIe card and three or four cables (depending on the disk enclosure) to talk to twelve SATA disks, and we could talk to more disks with a bigger disk enclosure. Without PMP, we could not even fit enough SATA connectors into the available physical space for PCIe card edges.

(SATA port multiplication is generally abbreviated PMP, and I'm going to do so for the rest of this entry.)

Unlike SCSI or IDE, port multipliers require hardware support on both sides; you don't directly connect to multiple SATA drives. The computer needs to have a multiplier-capable chipset (with a driver that knows how to use it) and the disk enclosure needs its own de-multiplier daughter cards that splits the connections apart to individual disks. PMP is generally backwards compatible with plain SATA and on the computer side a PMP-capable chipset is perfectly happy to talk to individual disk drives. What happens with PMP disk enclosures is that if you talk to their SATA ports without doing PMP, you only see the first drive behind each PMP SATA port; all of the other drives on the port are invisible. This can happen either if you have a non-PMP chipset or your chipset is PMP-capable but you don't have driver support for it yet.

(In Linux, this first drive is the drive that is enumerated as drive 0.)

In theory PMP allows up to fifteen drives to be behind each port. I believe that actual SATA chips have lower limits, and this may vary between chipsets; I have read discussions that suggest that common PMP-enabled chipsets have limits of 5:1 or 4:1 multiplication (that is, only four or five drives visible per SATA port).

There is one potentially important consequence of having multiple disks on a single SATA port. If something happens on the port and the SATA link must be reset (drivers like doing this for error recovery), you will lose contact with all disks on the port for the duration of the reset. So far this has not caused us problems. Note that hotswapping disks seems to lead to a port reset in our current environment.

Quality of implementation issues for PMP disk enclosures vary significantly in at least two ways: how drives are split between ports and how SATA drive numbers map to physical drive slots. We have two different brands of 12-disk enclosures with our iSCSI backends. The first brand has four ports with 5, 5, 1, and 1 drives each respectively (so yes, the last two ports are not even PMP, which I suppose saved the maker a PMP de-multiplier card) and the 5-drive ports map to physical slots in the order '2 3 4 5 1'. The second brand has three ports with 4 drives each and the physical slots are mapped sequentially, which is what you want for sysadmin sanity.

(Fortunately we have far more of the second brand that the first brand and the disks in the first brand's enclosures have been very reliable. Both models are now out of production so there's no point in naming names.)

PS: do not expect these things to be documented for inexpensive disk enclosures (and they're basically all inexpensive). I recommend careful experimentation and mapping in the OS of your choice.