ZFS and crazy dates

Recently, some backups on our new Solaris x86 fileservers started failing with (GNU) tar reporting errors about:

Warning: Cannot stat: Value too large for defined data type

More alarmingly, even ls on the Solaris fileserver would report this (as an error). Fortunately, I remembered a recent discussion on the ZFS mailing list about this, to the effect that if you are using NFS, it turns out to be possible to create files with 'impossible' timestamps. Inspection showed that this was indeed the case (and a pool scrub showed no errors).

(In this case, 'impossible' meant timestamps with the (32-bit) high bit set. Over NFS, the interpretation varied; our 32-bit Linux machines saw times well before 1970, while our 64-bit machines saw times well into the future.)

As mentioned in a message, the 64-bit versions of commands that are in /usr/bin/amd64 do work; it is only 32-bit ls and so on that have problems, presumably because Solaris is helpfully not letting them see 32-bit time_t values that have the high bit set.

(Everything in /usr/bin is 32-bit because x86 Solaris is, to put it one way, basically a 32-bit system with a bit of 64-bit stuff layered on top even on 64-bit installs. Personally I prefer real 64-bit installs.)

The selection of 64-bit commands in /usr/bin/amd64 is very odd and rather incomplete; for example, there is ls but not find (which also has the problem, especially since /usr/lib/fs/nfs/nfsfind will normally sweep all your local filesystems every so often). There is a 64-bit version of Python (which I used to determine the exact file timestamp) but not a 64-bit version of Perl.

Our Amanda backups were running into these problems because they were using a version of GNU tar that was (of course) compiled for 32-bit Solaris. I expect that we'll be working out how to change that soon (hopefully it will be as simple as rebuilding our local Amanda install with -m64).

(This is the sort of entry that I write partly as a note to myself.)