OmniOS boot environments are in general pretty cool things, but they do create one potential worry: how much data gets captured in them and thus how much space they can consume over time. Since boot environments are ultimately ZFS snapshots and clones, the amount of space each individual one uses over time is partly a function of how much of the data captured changes over time. Taking an extreme case, if you have a very large /var/log that is full of churning logs for some reason, each boot environment and BE snapshot you have will probably wind up with its own unique copy of all of this data.

(Unchanging data is free since it's shared between all BEs and BE snapshots.)

One of the things that this can push you towards is limiting what's included in your boot environments by making some things into separate filesystems. One obvious candidate here is /opt, where you may wind up with any number of local and semi-local packages that you update and otherwise churn at a much faster rate than base OmniOS updates and upgrades. After all this is the entire point of the OmniOS KYSTY principle and the historical use of /opt.

Well, I'll cut to the chase: don't do this if you want to be able to do upgrades between OmniOS releases, at least right now. You can create separate ZFS filesystems under bits of /opt, but if you take the obvious route of making all of /opt its own ZFS filesystem things will go terribly wrong. The problem is that some core OmniOS packages that you may wind up installing (such as their GCC packages) are put into /opt but upgraded as part of making a new boot environment on major upgrades. Because a boot environment only contains things directly in /, this doesn't work too well; pkg tries to update things that aren't actually there and will either fail outright or create things in /opt in the root of your new BE, which blocks mounting the real /opt.

I will summarize the resulting OmniOS mailing list discussion as 'a separate /opt is not a supported configuration; don't do that'. At the best pkg may some day report an explicit error, so that if you're stuck in this situation you'll know and you can temporarily remove all of those OmniOS packages in /opt.

(Our solution is to abandon plans to upgrade machines from r151010 to r151014. We'll reinstall from scratch and this time we'll make the largest single piece of /opt into a filesystem instead.)

My personal view is that this means that you do not want to build or install anything in /opt. Make up your own hierarchy, maybe /local, and use that instead; that should always be safe to make into its own filesystem. OmniOS effectively owns /opt and so you should stay out.

I believe that this is a general issue with all Illumos derived distributions if they put any of their own packages in /opt, such as GCC. I have not looked at anything other than OmniOS. I don't know if it's an issue on Solaris 11; I'd like to hope not, but then I have low confidence in Oracle getting this right either.

(You may think that being concerned about disk space is so 00s, in this day of massively large hard drives. Well, inexpensive SSDs are not yet massively large and they're what we like to use as root drives these days. They're especially not large in the face of crash dumps, where OmniOS already wants a bunch of space.)