Wandering Thoughts archives

Sun flubs another SSH patch

I haven't been involved with Solaris 9 for a while now, so I have no idea if they've fixed the Solaris 9 SSH patch problem by now (although I certainly hope they have). But I was recently heartened to discover that Sun is perfectly capable of fumbling Solaris 10 SSH patches as well.

Sun recently released patch 126134-03, 'sshd patch', for Solaris 10 x86, in order to fix CVE-2008-1483, where the fun semantics of IPv6 versus IPv4 let an attacker hijack forwarded X sessions. Unfortunately if you install this patch on a system without IPv6 enabled, you lose the ability to forward X at all.

(Instead, sshd syslogs the message error: Failed to allocate internet-domain X11 display socket. What it really means is that it failed to allocate any IPv6 listening sockets because you don't have IPv6 enabled, and it refuses to fall back to IPv4.)

It is difficult for me to understand how Sun managed to screw this one up. The bug is not unique to Sun's version of SSH and other operating systems managed to get the fix correct, and the problem is uncovered in literally ten seconds of testing on one of the most common customer configurations of Solaris 10 x86. Somehow Sun either let it slip through anyways or decided that it didn't matter and they would release the patch with known issues and without any sort of warning (and I am not sure which option would be worse).

Unfortunately, there's a bigger issue here than Sun continuing their history of screwing up ssh; it is that this makes it clear that you cannot trust Sun security patches. Untrustworthy security patches are only slightly better than no security patches, and arguably they're worse; at least with no security patches you know clearly where you stand.

PS: if you want workarounds, the ones here might work.

The problem with ZFS, SANs, and failover

The fundamental problem with Solaris 10's current version of ZFS in a SAN failover environment is that it has no concept of locking or host ownership of ZFS pools; instead, ZFS pools are either active (imported) or inactive (exported). So, if a host crashes and you want to fail over its ZFS pools, the pools are still marked as active, which means you must force pool import, which has catastrophic consequences if something ever goes wrong.

But it gets worse. Because hosts don't normally deactivate their pools when they shut down, a booting host will grab all of the pools that it thinks it should have regardless of their active versus inactive status and thus regardless of whether they are being used by another machine, because it cannot tell.

You can set ZFS pools so that they aren't imported automatically on boot (by using 'zpool import -R / ...'). However, in our iSCSI SAN environment each zpool import takes approximately a third of a second per LUN; a third of a second per LUN times a bunch of LUNs times a bunch of pools is an infeasibly long amount of time.

(And before you ask, as far as I can tell there is no opportunity to do something clever before Solaris auto-imports all of the pools during boot because the auto-importation happens by special magic.)

The conclusion is that if a host crashes and you want to fail over its pools, you must make utterly sure that it will never spontaneously reappear. (I recommend going down to the machine room and pulling its power and network cables and removing its disks. Make sure you zero them before you return them to any spares pool you have.)

If you try hard enough there are ways around some of this, such as storage fencing, where you arrange with your backend so that each host in the SAN can only see the storage with the pools that it should be importing. But this is going to complicate your SAN and your failover, and again if anything ever goes wrong you will have catastrophic explosions.

(Much of this is fixed in things currently scheduled for Solaris 10 update 6. Unfortunately we need to start deploying our new fileserver environment before then.)

My problem with ZFS

The more I use ZFS the less happy I become with it. It's not from any issues with ZFS's features or with how it works in practice; ZFS isn't one of those things that look better in theory than in practice. Based on my exposure so far it pretty much does live up to its promise, and the ZFS people are busy sanding off the rough edges and adding useful features.

My problem with ZFS is that if something goes wrong it all too often goes wrong in a big way; there are very few states between 'your pool is fine' and 'your pool is totally dead and probably panicing the system'. In practice, ordinary filesystems almost always degrade gradually, so that when something goes wrong (even fairly severe things) the system keeps lurching on as best as it can and keeps giving you as much data as possible. In the same sort of situations ZFS just shatters; your data is entirely lost and your entire system probably just went down. This might be tolerable if things only went badly wrong once in a while, but this is not the case; it is distressingly easy to run into situations that ZFS doesn't know how to cope with (I seem to have a talent for it).

So what about ZFS's vaunted resilience? My strong impression is that ZFS is only resilient against very limited sorts of damage; outside of those areas, ZFS makes no attempt to cope and just gives up entirely. This is no doubt easy to code, but does not strike me as at all the right thing to do.

This is especially bad because ZFS has no usable tools for recovering data from damaged filesystems, no equivalents of fsck and dump. With traditional filesystems, even badly damaged ones, you at least have tools that you can use to try to salvage something. With ZFS, damaged is dead; restore from backups.

(And ZFS snapshots will not help you, because those are stored within the ZFS pool; lose the pool and you lose the snapshots as well.)

Or in short: if something goes wrong with ZFS, it generally goes badly wrong. And things go wrong with ZFS too often for real comfort.

Sidebar: what went wrong today

I was testing what would happen if a ZFS pool on shared ISCSI storage was accidentally active on two systems at once, for example if you had a fumbled failover situation of some sort. So I did:

• import the pool on both systems (forcing it on the second one)
• read a 10 Gb test file on both systems
• export the pool on one system, ending that system's use of it
• scrub the pool on the other system to see what happened

What I hoped to happen was nothing much. What I got was not just a dead pool but a pool that paniced any system that tried to import it.