A basic overview of SAS and using SATA with SAS

September 14, 2013

I have been busy having a somewhat painful learning experience about SAS, especially bearing on putting SATA disks behind SAS. In my usual way I'm going to write down what I've learned so that it will stick.

SAS is yet another disk interconnect technology. It came out of SCSI (as opposed to SATA coming out of ATA) and so it is generally more large-scale, enterprisey, and costly. You can, at least in theory, use SATA disks in (some) SAS systems.

(For my purposes here I'm going to skip over the actual and claimed differences between SAS disks and SATA disks of the same basic specification. You can find partisans on both sides and I lack the knowledge to have an informed opinion.)

Like SATA, SAS is fundamentally a directly connected point to point system; one SAS port, one disk. However normal SAS connectors are 'multi-lane', which is to say that they have the actual wiring for several SAS ports bundled into one physical cable and connector. The essentially standard sorts of connectors (such as 'IPASS' aka SFF-8087) has four lanes and so one connector on your SAS card or SAS-equipped motherboard can connect to four drives (through various things in the middle to break out the lanes). SAS also has standard support for SAS expanders, which are very roughly the equivalent of SATA port multipliers except that they support many more drives and generally work better.

Many cases that support SAS drives (such as the SuperMicro SC836 series) have a SAS backplane that sits between the drives and the rest of the world (such as your server motherboard). There are at least three general ways that such a backplane can work: it can have one or more SAS expanders, it can wire all drives through with completely separate connectors (aka '1:1'), or it can have IPASS connectors on the outside and break out each lane to a drive on the inside (aka '4:1'). If the backplane has SAS expanders you need only a few SAS ports to talk to all of the disks (often only one or maybe two). Otherwise you need as many SAS ports as you have drives.

(I think that a JBOD SAS disk enclosure is almost always going to have a SAS expander or two so that you only have to connect up one or two external cables. A server chassis may use any of the three options and your vendor may even let you choose. A full discussion of which one you might want when is beyond both my experience and the scope of this entry.)

You can plug SATA drives into SAS drive connectors (although not vice versa). While the two sorts of drives use completely different protocols, a properly functioning SAS environment will arrange to speak SATA to SATA drives. If you are not using a SAS expander I believe that the host SAS controller does this directly. If you are using a SAS expander there is STP (the SATA Tunneled Protocol) and I believe that your host SAS controller talks STP to the SAS expander, which turns it into SATA when it talks to the drive(s). On the Internet you can read all sorts of bad things about using SATA drives in SAS systems but my impression is that what creates most of this badness is putting SATA drives behind SAS expanders (see eg this). SATA drives that are directly connected to host SAS controllers seem to be reported as working okay.

As far as I can tell this means that JBOD SAS disk enclosures with SATA disks are probably not recommended but SAS server cases may be okay depending on what their SAS backplane is. You want a non-expander option, which will probably have the inconvenient side effect that you'll need an additional SAS controller card or two (8-port cards seem to be the standard). This may have implications for your long term storage expansion plans.

(SAS disks remain sufficiently more expensive than SATA disks that we can't deal with the whole mess by just using SAS disks.)


Comments on this page:

By -dsr- at 2013-09-14 06:34:41:

For what's it's worth, I have several machines running with HP P800 SAS RAID cards attached to HP's MSA60 12x3.5" SAS chassis -- full of decent commercial SATA disks. No problems have occurred in this configuration. The MSA60 has in and out SAS channels (and dual power supplies) and you can daisy chain them if you're willing to accept that the same bandwidth now needs to feed more drives. Internally, it's a SAS expander.

I also have some machines with internal SAS expander backplanes, connected to 3Ware cards, and they do just fine with a mix of SATA SSDs and HDs.

STP is the name for the protocol conversion that happens on the Expander (or HBA in direct attach) that is used to translate between SATA and SAS. The signalling is different as well as the protocol itself which differs. It can be sufficiently mapped and there are passthrough commands that can send ATA commands to the disk if necessary (rarely so).

I worked quite a bit with SATA drives in a SAS backend through expanders and while we had problems and some were attributed to this mix I don't believe there is a real claim there. This might be different with different vendors, the quality of the STP implementation matters, a lot. More problems stem from bad assumptions about the disk behavior in the RAID then anything else.

An additional different is that SAS drives can be dual-ported, they may have two different ports on them so that they can be controlled independently from two initiators, something that SATA cannot do. It probably only matters to specific high availability environments (SAN devices), you could multi-path to the disk through the same JBOD in two different cables and get to the same drive through two different paths. There is still a SPOF in the JBOD mid-plane though.

The main reason I like SAS drives better than SATA drives is their error handling. They work pretty much the same as long as everything is fine but it is slightly better to diagnose an ailing SAS drive and its error handling in case of media-errors in superior and offers a fine-grained control over it.

As for prices, 7200 RPM drives are priced pretty similarly at local stores. Even in the same series (Seagate Constellation SAS vs SATA, personal preference is against Seagate but I don't have appropriate statistics anymore).

-- Baruch Even

Just did a quick Amazon search to verify: http://www.amazon.com/s/ref=sr_nr_p_n_feature_keywords_4?rh=n%3A1254762011%2Ck%3Ahdd+constellation%2Cp_n_feature_keywords_four_browse-bin%3A5054055011&keywords=hdd+constellation&ie=UTF8&qid=1379170233&rnid=5054030011

If the link doesn't work, search for "hdd constellation" 2 TB disks. They have both SAS and SATA versions. The SATA one is $182 and the SAS is $194. Not a big difference in my book.

The big difference usually is if you need to also buy a SAS Controller, but if you already have a SAS JBOD/Server the disks themselves are not a big difference.

By cks at 2013-09-14 22:38:57:

The best prices I can see for SAS drives of the sort we're looking at (2TB 7200 RPM) appear significantly more than the price we've been quoted for decently good 2TB SATA drives of the same specification (at the moment, WD SEs). They seem to cost about 20% to 25% more which is well over our budget.

(We're likely buying close to a hundred disks, so moderate price differences add up fast.)

By cks at 2013-09-16 12:59:02:

Belatedly, another note: my impression from my reading is that SATA disks behind SAS expanders generally work fine in ordinary situations. It's when you get into extraordinary ones that you see issues and problems, ie things like disks starting to die, disks freaking out over internal errors, really heavy IO load combined with unusual SATA operations, and so on.

(Intuitively this makes sense. If things exploded routinely everyone would know about it, but unusual situations and corner cases are rarely encountered unless you operate at (some degree of) scale.)

Mind you, I don't know if this is selective reporting and superstition. The people who aren't having problems with their systems are unlikely to show up in stackoverflow threads and so on.

By Baruch at 2013-09-17 16:33:48:

I obviously cannot disclaim other peoples problems and have just today went through a bug report on Linux mpt2sas driver that has consistent problems when connecting SATA drives to a SAS controller (I believe directly without an expander in the middle) and he has problems under high loads of reads.

My only valid claim is that I used sas controllers from LSI with an expander from PMC-Sierra and SATA drives (of various sorts, mostly Seagate and Hitachi) and had not even a single problem that I could attribute to the SATL. This was when I oversaw the platform/hardware of a storage vendor with thousands of servers so it was at a large scale and with a lot of different workloads, easily reaching into high workloads (raid rebuilds make it easy to get there).

It was though a single mix of hardware and I'm pretty sure that there are bad implementations of SATL that will not withstand the load or that will do something bad under duress.

The environments I worked in changed from SATA disks to SAS and the rate of disk failures stayed about the same, figuring out what failed and early detection of failures improved.

Written on 14 September 2013.
« Why I think dir() excludes metaclass methods
Identities, trust, and work »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Sep 14 01:15:39 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.