Wandering Thoughts archives

Installing and pinning specific versions of OmniOS packages

Suppose, not entirely hypothetically, that you want to install some additional OmniOS systems with their kernels pinned to a specific version, one that you know works because it's what you're already running on your current OmniOS machine (in fact, perhaps you are having problems with a more recent kernel on your test machine). In order to do this, you're going to need to do three things: you need to find out what versions of what packages to pin, then you need to actually install those specific versions of the packages, and finally you need to keep them from changing.

Once you have the full FMRIs of the packages you want to have a specific version of when you install a new system, there are two cases: install time upgrades and downgrades. The simplest approach is upgrades; you start from an old OmniOS installer image for your OmniOS version, then upgrade the packages you need to only their specific version instead of all the way to the latest one. This is done by giving 'pkg update' the full package FMRIs on the command line; if you have a file called pkg-versions, this is:

pkg update $(cat pkg-versions)

But wait, as they say, there is a complication (at least for the kernel). Kernel upgrades require a new boot environment (and are done in that new BE) and if you're installing from an older image, you have other packages to upgrade too and some of them may also require a new boot environment. So what you really want to do is install both your specific versions and 'the latest updates to other packages' at the same time. This is done by also giving 'pkg update' an explicit "*" argument:

pkg update $(cat pkg-versions) "*"

This lets you do the whole process in a single new BE and a single reboot.

If you're installing from the very latest OmniOS installer image for your OmniOS version, you actually need to downgrade your packages. According to the documentation this is also done with 'pkg update $(cat pkg-versions)', but I haven't tested it so I don't know if it works right. My moral is save your old OmniOS installer images, partly because I trust 'upgrade only to a specific version' more than 'start with a more recent version and downgrade'

(It's possible that old installer images are still available somewhere, but I don't know where to find them. Old package versions are kept around in at least the r151014 repo and this will hopefully continue to be the case.)

Once you've actually installed your versions of the specific packages, you need to freeze them against further upgrades. This is done with the straightforward 'pkg freeze <FMRI> [...]':

pkg freeze -c "known to work kernel" $(cat pkg-versions)

However, once again boot environments get in the way. Installing our kernel packages created a new boot environment, and you have to freeze packages in the new BE, not the current one. So the easy way to go is to reboot into the new BE before you run the 'pkg freeze'.

(It's possible that you could manually mount the new BE with 'beadm mount' and then point 'pkg freeze' at it with pkg's -R argument, per here. However I haven't tested it and I honestly think it's probably simpler to reboot into the new BE first under most circumstances.)

PS: It's unfortunate that 'pkg freeze' won't freeze specific versions that are later than what you have installed. Otherwise the easy approach would be to freeze first, run a general 'pkg upgrade' (which should upgrade your packages to their frozen versions and everything else to the latest versions), and then reboot into your new BE. But that would probably make freezing more complex in the pkg code, so I can sort of see why it isn't allowed.

Some notes on finding package versions in OmniOS with pkg

For reasons that don't fit within the margins of this entry, I recently had to poke around the pkg system on OmniOS in order to find out some information about packages, such as which package versions are available in the OmniOS repo, what package versions are on the system, and what packages provide certain files.

So, first, versions. pkg packages have a short version and a long one, as so:

$ pkg list kernel
NAME (PUBLISHER)  VERSION
system/kernel     0.5.11-0.151014
$ pkg list -v kernel
FMRI
pkg://omnios/system/kernel@0.5.11-0.151014:20150929T225337Z

(In all my examples, some output is condensed and fields omitted.)

As you might guess from the format of the short version, all kernel packages for OmniOS r151014 have the same short version; they differ only in the timestamp on the long version. This means that if you care about the specific kernel version for some reason you must ask for the long version.

The OmniOS r151014 repo has (at least right now) all kernel versions published for r151014, from the start onwards. You can see all of the available versions with 'pkg list -afv kernel':

$ pkg list -afv kernel
FMRI
pkg://omnios/system/kernel@0.5.11-0.151014:20150929T225337Z
pkg://omnios/system/kernel@0.5.11-0.151014:20150914T195008Z
[...]
pkg://omnios/system/kernel@0.5.11-0.151014:20150402T175237Z

If for some reason you want to install an older kernel, this is what you may need to do to find out its specific full version.

Now, the OmniOS kernel is not delivered in just the kernel package; instead there are a whole collection of packages that contain kernel drivers and other modules. So if you want 'a specific older kernel', you probably want not just the basic kernel package but all of the related drivers to be from that older kernel. This leads to the question of what installed packages on your system supply kernel drivers, and for that we turn to 'pkg contents'. To get a list of all such files along with the package names of the packages that supply them, we want:

$ pkg contents -t file -o path,pkg.name -a 'path=kernel/*'
PATH                    PKG.NAME
kernel/amd64/genunix    system/kernel
[...]
kernel/drv/aac          driver/storage/aac
[...]
kernel/drv/amd64/fct    storage/stmf
[...]
kernel/drv/amd64/fm     service/fault-management
[...]
kernel/drv/amd64/iscsi  network/iscsi/initiator
[...]
kernel/drv/amd64/zfs    system/file-system/zfs
[...]
kernel/fs/amd64/nfs     system/file-system/nfs
[...]
kernel/kmdb/amd64/arp   developer/debug/mdb
[...]

(To get long versions, ask for pkg.fmri instead of pkg.name. I've used short names because this example is already long enough.)

As this rather long example shows, packages from all over the package namespace can wind up providing kernel modules; they are by no means confined to driver/* and system/kernel* as you might innocently initially expect (although those certainly have the majority of kernel-related packages). You might wonder if the versions of all of these packages are tightly tied together so that they must be installed or updated as a set. As far as I know, the answer is that they (mostly?) aren't, apparently because Illumos has a stable kernel module API and most or all kernel modules use it. Whether or not the result works really well is an open question, but the package system itself won't prevent a mix and match situation in my brief testing.

To get just the package names (or FMRIs), we just need the second field like so:

$ pkg contents -t file -H -o path,pkg.fmri -a 'path=kernel/*' |
      awk '{print $2}' | sort -u

This will give us a nice list of specific package versions that are responsible for files under /kernel in our current system.

However, suppose that we've recently updated to the latest r151014 update but the new kernel may have problems in our testing, and what we'd really like to get is the versions of the last kernel. Since a kernel update makes a new boot environment, one option is to just reboot into the old pre-update boot environment and run these 'pkg contents' or 'pkg list' commands. But that might be disruptive to ongoing tests and it turns out that we don't need to, because we can make pkg look at alternate boot environments (although not directly).

First we need to know what boot environments we have:

# beadm list
BE                Active [...]
[...]
omnios-cslab2-2   -
omnios-cslab2-3   NR

Assuming that we want the obvious previous BE, now we need to mount it somewhere:

# beadm -s ro omnios-cslab2-2 /mnt

Now we can look at package information for this old BE by giving pkg the -R option, for example:

$ pkg -R /mnt contents [...]

When you're done, unmount the BE with 'beadm umount'.

This provides a handy and relatively non-intrusive way to recover specific package versions from an old boot environment (or, for that matter, just a list of installed packages).

Sidebar: My sources for this

Some of this is derived from information in the OmniOS wiki's general administration guide. It gives a number of 'pkg contents' examples that were quite helpful. In general 'pkg contents' can be used to do all sorts of things, it's not at all limited to mapping files to packages and packages to files.

Information about 'pkg list' and pointing pkg at alternate BEs is from Lauri Tirkkonen on omnios-discuss in answer to me asking how to do this sort of stuff.

How much space ZFS reserves in your pools varies across versions

Back in my entry on the difference in available pool space between zfs list and zpool list, I noted that one of the reasons the two differ is that ZFS reserves some amount of space internally. At the time I wrote that the code said it should be reserving 1/32nd of the pool size (and still allow some things down to 1/64th of the pool, like ZFS property changes) but our OmniOS fileservers seemed to be only reserving 1/64th of the space (and imposing a hard limit at that point). It turns out that this discrepancy has a simple explanation: ZFS has changed its behavior over time.

This change is Illumos issue 4951, 'ZFS administrative commands should use reserved space, not fail with ENOSPC', which landed in roughly July of 2014. When I wrote my original entry in late 2014 I looked at the latest Illumos source code at the time and so saw this change, but of course our ZFS fileservers were using a version of OmniOS that predated the change and so were using the old 1/64th of the pool hard limit.

The change has propagated into various Illumos distributions and other ZFS implementations at different points. In OmniOS it's in up to date versions of the r151012 and r151014 releases, but not in r151010 and earlier. In ZFS on Linux, it landed in the 0.6.5 release and was not in 0.6.4. In FreeBSD, this change is definitely in -current (and appears to have arrived very close to when it did in Illumos), but it postdates 10.0's release and I think arrived in 10.1.0.

This change has an important consequence: when you update across this change, your pools will effectively shrink, because you'll go from ZFS reserving 1/64th of their space to reserving 1/32nd of their space. If your pools have lots of space, well, this isn't a problem. If your pools have only some space, your users may notice it suddenly shrinking a certain amount (some of our pools will lose half their free space if we don't expand them). And if your pools are sufficiently close to full, they will instantly become over-full and you'll have to delete things to free up space (or expand the pool on the spot).

I believe that you can revert to the old 1/64th limit if you really want to, but unfortunately it's a global setting so you can't do it selectively for some pools while leaving others at the default 1/32nd limit. Thus, if you have to do this you might want to do so only temporarily in order to buy time while you clean up or expand pools.

(Of course, by now most people may have already dealt with this. We're a bit behind the times in terms of what OmniOS version we're using.)

Sidebar: My lesson learned here

The lesson I've learned from this is that I should probably stop reflexively reading code from the Illumos master repo and instead read the OmniOS code for the branch we're using. Going straight to the current 'master' version is a habit I got into in the OpenSolaris days, when there simply was no source tree that corresponded to the Solaris 10 update whatever that we were running. But these days that's no longer the case and I can read pretty much the real source code for what's running on our fileservers. And I should, just to avoid this sort of confusion.

(Perhaps going to the master source and then getting confused was a good thing in this case, since it's made me familiar with the new state of affairs too. But it won't always go so nicely.)