ZFS versus SANs: where do you put the RAID?
Here is an issue I have been thinking about recently: how do you want to handle ZFS in a SAN environment, especially one in with relatively low data rates such as, oh, anything based on gigabit Ethernet?
The problem is that ZFS and SANs have a small conflict. ZFS wants to talk to raw disks and do RAID internally so that it can do its special magic, but SANs want you to let the backends do as much RAID work as possible to preserve overall bandwidth. If you do mirroring in ZFS, for example, you have just doubled the write bandwidth that you need.
(Read bandwidth should be mostly unaffected, except for your periodic integrity scans. You do do periodic integrity scans, right?)
Various forms of link aggregation and multipathing can give you more iSCSI bandwidth, but my impression so far is that you have to get lucky to have both ends support the same sort of aggregation. It also means a bigger switch infrastructure and at some point you may run into limits on the total bandwidth your switches can support.
How much you have to worry about this depends on how much write bandwidth you need (and on what sort of RAID you'd have ZFS do; mirroring adds more write IO than RAID 6, which has more than RAID 5). If you fall in the middle, with not so little write IO that you can ignore this and not so much write IO that you have no choice, this seems clearly a 'pick your poison' question; both options have disadvantages.
(Here, the winning argument is likely to be that if we let ZFS do the mirroring and always mirror between two SAN backends, our machines may not reboot if they can't talk to one briefly.)