Don't make /opt a filesystem on OmniOS (or probably Illumos generally)

April 22, 2015

OmniOS boot environments are in general pretty cool things, but they do create one potential worry: how much data gets captured in them and thus how much space they can consume over time. Since boot environments are ultimately ZFS snapshots and clones, the amount of space each individual one uses over time is partly a function of how much of the data captured changes over time. Taking an extreme case, if you have a very large /var/log that is full of churning logs for some reason, each boot environment and BE snapshot you have will probably wind up with its own unique copy of all of this data.

(Unchanging data is free since it's shared between all BEs and BE snapshots.)

One of the things that this can push you towards is limiting what's included in your boot environments by making some things into separate filesystems. One obvious candidate here is /opt, where you may wind up with any number of local and semi-local packages that you update and otherwise churn at a much faster rate than base OmniOS updates and upgrades. After all this is the entire point of the OmniOS KYSTY principle and the historical use of /opt.

Well, I'll cut to the chase: don't do this if you want to be able to do upgrades between OmniOS releases, at least right now. You can create separate ZFS filesystems under bits of /opt, but if you take the obvious route of making all of /opt its own ZFS filesystem things will go terribly wrong. The problem is that some core OmniOS packages that you may wind up installing (such as their GCC packages) are put into /opt but upgraded as part of making a new boot environment on major upgrades. Because a boot environment only contains things directly in /, this doesn't work too well; pkg tries to update things that aren't actually there and will either fail outright or create things in /opt in the root of your new BE, which blocks mounting the real /opt.

I will summarize the resulting OmniOS mailing list discussion as 'a separate /opt is not a supported configuration; don't do that'. At the best pkg may some day report an explicit error, so that if you're stuck in this situation you'll know and you can temporarily remove all of those OmniOS packages in /opt.

(Our solution is to abandon plans to upgrade machines from r151010 to r151014. We'll reinstall from scratch and this time we'll make the largest single piece of /opt into a filesystem instead.)

My personal view is that this means that you do not want to build or install anything in /opt. Make up your own hierarchy, maybe /local, and use that instead; that should always be safe to make into its own filesystem. OmniOS effectively owns /opt and so you should stay out.

I believe that this is a general issue with all Illumos derived distributions if they put any of their own packages in /opt, such as GCC. I have not looked at anything other than OmniOS. I don't know if it's an issue on Solaris 11; I'd like to hope not, but then I have low confidence in Oracle getting this right either.

(You may think that being concerned about disk space is so 00s, in this day of massively large hard drives. Well, inexpensive SSDs are not yet massively large and they're what we like to use as root drives these days. They're especially not large in the face of crash dumps, where OmniOS already wants a bunch of space.)

Comments on this page:

By lotheac at 2015-04-22 03:06:16:

(Our solution is to abandon plans to upgrade machines from r151010 to r151014. We'll reinstall from scratch and this time we'll make the largest single piece of /opt into a filesystem instead.)

I don't think you need to reinstall for this if your problem is having packages installed in /opt. I would use beadm to create and mount a new BE, making sure not to mount the /opt fs under it, and then run 'pkg -R $mntpnt fix' to create the missing files and 'pkg -R $mntpnt update' to upgrade afterwards.

By cks at 2015-04-23 01:09:39:

I wasn't clear in my brief description. The problem is not that the new BE has /opt mounted but that it doesn't, and the BE needs /opt mounted in order to successfully update package with files in /opt. If you 'fix' the now-missing package files in a BE that lacks a mounted /opt, the upgrade will work but ZFS will refuse to mount your real /opt when you boot into the BE because the /opt directory on the root filesystem has contents (namely those packages that got repaired in it).

Written on 22 April 2015.
« I don't think I'm interested in containers
Upgrading machines versus reinstalling them »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Apr 22 01:03:38 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.