Some thoughts on battery backup for RAID controller cards
In a comment on my entry on software RAID's advantages I was asked what I thought about sofware RAID's lack of battery backup units, as you can get on better RAID controller cards. To answer that, I'm going to start by asking my traditional question: how does having a BBU RAID card improve your system performance?
A RAID card with a battery backup unit effectively turns synchronous disk writes into asynchronous ones, by buffering such writes in its battery-backed RAM and immediately telling the host OS that the write has completed successfully. In order for this to improve performance, you have to be doing enough synchronous writes to stall your system significantly. It helps if they're relatively slow synchronous writes; the classical case is small synchronous writes to a RAID-5 array, where the small OS-level write actually turns into a couple of reads and a couple of writes.
(There is also a limit to how much a BBU can improve your performance, especially your sustained performance; at a certain write load you hit your disk's performance limits, either in write bandwidth for streaming writes or in IO operations a second for random writes. If you need to do a sustained 500 random writes per second to a single physical disk, no BBU can help you.)
In general, most write workloads are mostly asynchronous; however, there are certainly some that are highly synchronous (database operations, mail servers, anything that does a lot of file write operations over NFS, etc). These days, operating systems are very good at not forcing synchronous writes unless they feel that they really have to, because filesystem authors fully understand that synchronous writes are death to performance. (Sometimes they go overboard in this.)
In exchange for this synchronous write acceleration, you accept a number of potential drawbacks. Obviously BBU RAID cards cost more, you have to use hardware RAID to some degree, the battery backup only lasts so long (although I believe it commonly lasts for days), and the RAID controller has to lie to the host OS about the write being successful. The latter may especially be an issue if you want to use the hardware RAID controller purely for JBOD-with-BBU, and do your actual RAID in software (such as with ZFS); there you would really like the OS level to find out about write errors.
These days, there are often higher-level options than BBU hardware RAID even if you have a lot of synchronous writes. For example, it's increasingly common for filesystems to let you put their logs on very fast disk storage (either SSDs or small fast conventional disks), and this can drastically accelerate synchronous filesystem writes.
(I was going to say that you could always put a UPS on the system as a whole, but that doesn't really solve the problem of synchronous writes unless you tell the operating system to lie about them.)
(Disclaimer: this is partly me thinking through this out loud. As I don't have actual experience with BBU hardware RAID, I could be completely off base on some of this.)
Sidebar: synchronous writes and us
On our workload of aggregate general fileservice, I don't think we have any particular density of system-stalling synchronous writes. While synchronous NFS activities can stall individual filesystems (and possibly individual ZFS pools, since I'm not entirely sure how the ZFS synchronous write process works), we have enough filesystems and ZFS pools and disks that the overall fileservers will continue going along without most people noticing anything.
As such, we don't worry about BBU issues, and in fact we deliberately configure our iSCSI backends to not have write caching, despite them being on UPSes and being considered reliable black boxes.