Intel has screwed up their DC S3500 SSDs

August 9, 2014

I ranted about this on Twitter a few days ago when we discovered it the hard way but I want to write it down here and then cover why what Intel did is a terrible idea. The simple and short version is this:

Intel switched one of their 'datacenter' SSDs from reporting 512 byte 'physical sectors' to reporting 4096 byte physical sectors in a firmware update.

Specifically, we have the Intel DC S3500 80 GB model in firmware versions D2010355 (512b sectors) and D2010370 (4K sectors). Nothing in the part labeling changed other than the firmware version. Some investigation since our initial discovery has turned up that the 0370 firmware apparently supports both sector sizes and is theoretically switchable between them, and this apparently applies to both the SC3500 and SC3700 series SSDs.

This is a really terrible idea that should never have passed even a basic smell test in a product that is theoretically aimed at 'datacenter' server operations. There are applications where 512b drives and 4K drives are not compatible; for example, in some ZFS pools you can't replace a 512b SSD with a 4K SSD. Creating incompatible drives with the same detailed part number is something that irritates system administrators a great deal and of course it completely ruins the day of people who are trying to have and maintain a spares pool.

This Intel decision is especially asinine because the 'physical sector size' that these SSDs are reporting is essentially arbitrary (as we see here, it is apparently firmware-settable). The actual flash memory itself is clumped together in much larger units in ways that are not well represented by 'physical sector size', which is one reason that all SSDs report whatever number is convenient here.

There may well be good reason to make SSDs report as 4k sector drives instead of 512b drives; if nothing else it is a small bit closer to reality. But having started out with the DC S3500 series reporting 512b sectors Intel should have kept them that way in their out of box state (and made available a utility to switch them to 4k). If Intel felt it absolutely had to change that for some unfathomable reason, it should have at least changed the detailed part number when it updated the firmware; then people maintaining a spares stock would at least have some sign that something was up.

(Hopefully other SSD vendors are not going to get it into their heads to do something this irritating and stupid.)

In related news we now have a number of OmniOS fileservers which we literally have no direct spare system disks for, because their current system SSDs are the 0355 firmware ones.

(Yes, we are working on fixing that situation.)


Comments on this page:

From 108.175.235.110 at 2014-08-09 09:14:55:

While it's probably too late for you, perhaps add an @Intel and/or #Intel to your tweet/s? The former is mostly a PR account, but it couldn't hurt as a PSA exercise for others.

I've never bothered FW-updating my SSDs unless I've heard about loss-of-data failures.

Does this FW update otherwise fix a real problem? I'd rather not touch my existing S3500s (especially after reading this) if I can help it.

By cks at 2014-08-09 17:29:58:

I haven't looked at the firmware release notes in any detail so I don't know what it covers. I also should have been clearer in my entry: we didn't update the firmware, we just got some drives with the 0355 firmware and other drives with the 0370 firmware (and then the drives we bought more recently as spares all came with the 0370 firmware). Thus it seems likely that even a few months ago when we bought the hardware from our VAR, Intel was busy flushing the 'old' 0355 drives out of their supply pipeline in favour of the 'new' 0370 drives.

As far as I know you can't get 0355 drives any more if you have to buy more DC S3500s. I also assume that if you RMA a dead 0355 drive under warranty you get a 0370 drive as the replacement; I'll know for sure when our RMA for our dead 0355 drive shows up.

By Lasse at 2014-08-13 15:43:07:

Any news on how to switch it? Is it not possible to downgrade the FW ?

- Lasse
By cks at 2014-08-13 16:11:04:

I doubt that Intel supports firmware downgrades, partly because it's quite possible that the new firmware is necessary due to internal hardware changes. We haven't checked because we don't consider it a really viable option.

We have heard that Intel has a utility that can change a drive's reported physical sector size (which would make sense, since in theory a switchable PSS is one of the improvements in the 0370 firmware), but we don't yet have a copy of such a thing if one exists.

By Paul Reid at 2014-11-03 21:34:19:

There is a tool that Intel support will provide that will allow you to switch the sector size back and forth.

They keep the tool private because it can lead to lost data - but when receiving a new SSD via RMA or purchase for installation into an existing array, the drive will be new and switching the sector size isn't a problem. Just don't do it on a drive with existing data on it, or you may lose that data.

Still, it's maddening that this even happened on a drive with the same part number.

I think Intel should just release the tool with massive warnings on it about loss of data if used on a drive with existing data, etc. Rather than deal with the communities backlash.

By anonymouse at 2014-11-18 19:21:44:

Mission critical lifetime hardware replacements are bought at the same time as the original hardware buy. Its pricey to do it that way, but think of long lifetime hardware deployments like the military. I hear they even have models to project the future lifetime hardware needs. You can't buy pit of manufacturing products 15yr later.

By TimC at 2014-11-18 20:44:31:

There's a very good reason. While the chips may be grouped in larger sizes, ALL SSD's use 4k or 8k pages for reads and writes (the vast majority in my experience use 4k pages). Presenting 512b sector sizes up to the OS results in a ton of wasted IOPS.

Written on 09 August 2014.
« Hardware can be weird, Intel 10G-T X540-AT2 edition
What I want out of a Linux SSD disk cache layer »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Aug 9 00:04:43 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.