Why the University of Toronto can't just use Google Mail
A lot of people like Google Mail, and it's not hard to see why; they have the best webmail interface, offer lots of storage, and so on. So why not just cut to the chase and make our users happy by outsourcing our email to Google Mail?
The answer in a nutshell is that we'd like a Google Mail Appliance, but the university is not in any position to make Google Mail our email provider.
There's a bunch of pragmatic reasons why outsourcing email would be hard to set up: we'd need a contract, a service level agreement, various features (and lack of features like Google Ads), a way to manage user accounts ourselves, and so on. But none of those are the real problem, because we could overcome them with money and negotiation (assuming Google was sufficiently interested).
The real issue is that both morally and legally, the University can't hand physical control of its private and internal email over to a third party, especially a third party located in America. Apart from the moral issues, these days we actually have a specific legal requirement to keep people's personal information private and, as part of that, in our direct custody.
(Direct custody is important because it is the only way that we can assert with a straight face that we even have a chance of knowing who everyone is who has access to the data.)
No service level agreement can fix this for two reasons. First, no SLA can overrule an American court order. And second, all that an SLA would do even in theory is let us collect some amount of money from Google, and money is often a poor compensation for a privacy breach.
(You might think that the court order issue is a small bagatelle. Think again; the University has a significant number of international students, and we take the associated issues quite seriously.)
A Google Mail Appliance wouldn't have these problems; Google would just be supplying the software (and the physical machine), and the email would stay under our physical and legal custody, in our machine room.
A disappointment with ZFS
According to the ZFS documentation for the current official Solaris 10 release (I don't know about the state of OpenSolaris builds, and they're not of interest to us anyways), you cannot make a storage pool that is a mirror of RAID-Z2 pools.
(You can indirectly construct a mirror of RAID-5 pools, by building the RAID-5 pools with the Solaris Volume Manager instead of ZFS itself.)
We want some form of RAID-6 instead of RAID-5 because RAID-6 leaves you less exposed to data loss than RAID-5 plus a hot spare for the same number of disks. The problem with RAID-5 is that on modern SATA disks, resyncing onto your hot spare after you lose a disk may take an appreciable amount of time; during this time, you're exposed to data loss if you lose a second drive. Since you're going to have the second parity drive anyways (in the form of a hot spare), you might as well have it live.
We want mirroring so we can mirror across two iSCSI controllers, so we can survive losing an entire controller without user downtime. (The price of iSCSI based SATA storage is low enough that we can actually afford to do this.)
And finally, we need ZFS to do the RAID-Z2 because our chosen iSCSI controllers don't have RAID-6, only RAID-5. Since we want mirroring on top of the RAID-Z2 from a single controller, this means ZFS has to provide it too. The inability to do it is disappointing; if we use ZFS, we will have to live with mirroring over controller-based RAID-5.
(Technically you might be able to get ZFS to do this by making two RAID-Z2 pools, making one big filesystem on each, making one big file on each filesystem, and then turning around and telling ZFS to make a mirrored storage pool using those two files. Somehow I don't think this is an officially supported configuration, though.)