Our sysadmin environment
The simple way to describe it is that my group is traditional university sysadmins (well, for departmental computing). This means that our job is to provide an environment with various services, things like network connectivity, reasonably decent file storage, printing, email, some machines that people can log into to do various things (including run big computational jobs), and a reasonably flexible web server that people can put stuff on. But that's pretty much it.
(What's in this central environment is mostly set by the department's computing committee, mostly based on what's easy and cheap to do and of sufficiently broad use to the department.)
If a graduate student wants something that we don't already provide, it's usually up to them (and their Point of Contact) to set it up, and if they have a program or a web service or whatever that is falling over or not performing well enough, it's entirely up to them to troubleshoot it. If a graduate student wants something that requires root permissions or a new system daemon or whatever, something they cannot set up as an ordinary Unix user, the answer is generally 'no, we can't provide that'. The escape hatch from this limited functionality is that people can put their own machines on the network and obviously do more or less whatever they want on them.
(We're willing to install standard Ubuntu packages for people, but we won't run new system daemons. So we'll install the PostgreSQL packages but not run a system PostgreSQL instance.)
This means that we do only one part of the traditional 'ops' jobs, and we do it in a very hands-off way. We don't get handed software to deploy, we don't get asked to set up various packages (eg, 'this thing needs an Oracle database, set one up for us'), and we don't even get new machines to install for people. We do have some custom software and systems that are part of our environment, but they're all stuff we've written and have full control over; we own them from start to finish, plus they're only providing internal services.
Even in a university, not all systems are run this way. There are plenty of important public-facing systems and services around here that are run in a much more conventional industry-like 'ops' way, with code and site deployments, the sysadmins being responsible for the service staying up, and so on. (Some of them lead to very interesting war stories and challenges.)
(Points of Contact probably have a more industry-like 'ops' job, but even there my impression is that graduate students usually retain a lot of responsibility for deploying their own software and keeping it running.)
Why I no longer believe that you need Solaris if you want ZFS
Four years ago I wrote an entry on why you wanted to use Solaris if you were going to use ZFS. Recently I have been reconsidering this issue, and I no longer believe that you need to pick Solaris if you're going to use ZFS. What has happened is that ZFS and ZFS development has changed drastically.
Back in 2008 it was clear that there was only one ZFS. All of the real ZFS development was happening at Sun and was being done to Solaris; all other versions were copying this work with various delays. Today in 2012 there's effectively not one ZFS any more, but instead at least two and maybe three (or more): Illumos ZFS, Solaris ZFS, and perhaps FreeBSD ZFS. (I don't know how separate FreeBSD ZFS is from Illumos ZFS.)
Illumos ZFS has real developer firepower behind it (many of the original
ZFS developers have left
Sun Oracle and moved to companies
that contribute to Illumos), while at the same time Oracle has made
changes that make Solaris 11 far less desirable (eg much higher costs
and closed source). It also seems likely that
neither version of ZFS will get really compelling changes (like the
ability to remove vdevs from a pool). This makes the two versions of
ZFS much more balanced and competitive, and the lack of major changes
makes a (potentially) older ZFS like FreeBSD's not that unattractive.
(As for support and bug fixes, let's just say that I expect even less from Oracle than from Sun.)
Another, less complementary way of putting it is that with ZFS today what you see now is pretty much what you're going to get in the future. Major changes might happen but they don't seem to be the way to bet. With ZFS basically frozen it's much easier to look at something like FreeBSD, evaluate its ZFS, and say 'this is good enough for us'; you're unlikely to be missing anything important in the future no matter what happens (or doesn't happen) with FreeBSD ZFS development.
To condense a potentially long discussion, all of this leaves me feeling that FreeBSD is now a generally viable mainline ZFS platform. It doesn't have the absolutely latest ZFS and bugfixes (whether you consider these to be the Illumos ones or the Solaris ones), but it has other advantages and its ZFS is likely to be good enough for most things.
(If you really need the features of Oracle Solaris's ZFS, even despite the uncertainties, well, you don't have a choice right now and maybe not ever. But I don't think many people are stuck like that, and I do mean 'stuck'.)