Wandering Thoughts archives

ZFS and crazy dates

Recently, some backups on our new Solaris x86 fileservers started failing with (GNU) tar reporting errors about:

Warning: Cannot stat: Value too large for defined data type

More alarmingly, even ls on the Solaris fileserver would report this (as an error). Fortunately, I remembered a recent discussion on the ZFS mailing list about this, to the effect that if you are using NFS, it turns out to be possible to create files with 'impossible' timestamps. Inspection showed that this was indeed the case (and a pool scrub showed no errors).

(In this case, 'impossible' meant timestamps with the (32-bit) high bit set. Over NFS, the interpretation varied; our 32-bit Linux machines saw times well before 1970, while our 64-bit machines saw times well into the future.)

As mentioned in a message, the 64-bit versions of commands that are in /usr/bin/amd64 do work; it is only 32-bit ls and so on that have problems, presumably because Solaris is helpfully not letting them see 32-bit time_t values that have the high bit set.

(Everything in /usr/bin is 32-bit because x86 Solaris is, to put it one way, basically a 32-bit system with a bit of 64-bit stuff layered on top even on 64-bit installs. Personally I prefer real 64-bit installs.)

The selection of 64-bit commands in /usr/bin/amd64 is very odd and rather incomplete; for example, there is ls but not find (which also has the problem, especially since /usr/lib/fs/nfs/nfsfind will normally sweep all your local filesystems every so often). There is a 64-bit version of Python (which I used to determine the exact file timestamp) but not a 64-bit version of Perl.

Our Amanda backups were running into these problems because they were using a version of GNU tar that was (of course) compiled for 32-bit Solaris. I expect that we'll be working out how to change that soon (hopefully it will be as simple as rebuilding our local Amanda install with -m64).

(This is the sort of entry that I write partly as a note to myself.)

My view of Sun and their history

From its founding in the the 1980s through the early 1990s, Sun was a powerhouse that delivered both a stream of technical innovations and an evolving range of machines, from small workstations through large servers, that were excellent values. While never quite as sexy as SGI (even if they sometimes tried hard), Sun's workstations were something that you could dream of having on your desk even if you weren't a Hollywood SFX worker, a CAD/CAM operator, or someone with supercomputer results to visualize. And their servers were the sensible default choice for basically everything, from small to large.

The wheels came off the wagon in the mid 1990s, starting when the features of Intel PCs began making everyone's low-end workstations look dowdy. Sun did not really react in any effective way to the relentless march of increasingly featureful PCs and Intel's increasingly capable CPUs. While Sun made abortive, half-hearted, and overpriced attempts to have some x86 products, it was very clear that their heart wasn't in it.

(Even when SPARC held the performance edge, for basic workstations it rankled that everything else was worse than on a PClone; you got worse graphics, worse serial speeds, proprietary keyboards, and so on.)

By the late 90s Sun and the SPARC architecture was no longer competitive with x86 hardware; Sun machines were invariably overpriced and had less features than comparable PClones, where there was anything comparable. From my perspective as a potential customer, Sun's reaction was to trade on their past reputation (essentially demanding a significant premium for the Sun name on a machine) and their very large servers, where they still had no x86 competition.

(In 1999 we evaluated new workstations for an undergrad lab that had previously been using X terminals and low-end SGI machines. There was nothing from Sun that was even vaguely competitive with PCs; that the Ultra series was then using PC components just made the cost, feature, and performance differences more pathetic and embarrassing.)

In a demonstration of the advantages of good marketing and sales, this strategy worked well enough to keep Sun in business. However, it did not endear them to me; from my perspective, the old Sun (the one that delivered value) had mutated into a marketing driven organization that was playing people for suckers. On the technical side, Sun seemed to be stagnating or at least ineffectual, unable to deliver anything particularly impressive that would justify Solaris over one of the free alternatives; Solaris was left to be trumpeted as merely 'more reliable'.

Starting sometime in the early 00s, Sun (the old Sun) woke up. It started making competitively priced and quite attractive generic x86 hardware, it made a real commitment to x86 in general, and it eventually did something novel with SPARC. It continues to be mired in old habits, but there's some excuse for that. In short, it is once again an exciting company, although one with some old baggage (just like every company).

I don't think that Sun is unusual for following this trajectory (except the last bit); all of the old Unix vendors followed very similar paths. Sun is just unusual in surviving, which I think is at least partly due to how strong it was up through the early 90s (and I suspect that Java didn't hurt). Similarly, I don't think that Sun has made more than the usual number of moves that are inexplicable or mistakes. Even the nadir of their late 90s period was commercially successful, just irritating.

(And it is not as if riding your brand into the ground is uncommon; my impression is that there are all sorts of reasons for the sort of organization paralysis and denial that causes that. The novelty is companies that manage to pull themselves out of the situation.)

A thesis: Sun should be prepared to give up on SPARC

One of the things that Sun is notable for is for basically being the last general computer vendor standing with a non-x86 architecture. Without getting into the debate about the SPARC architecture's long and somewhat questionable history, I think that Sun now really needs to be prepared to give up on SPARC.

First, let's admit something. In general, SPARC is on its last legs, as it has already lost out on performance and general desirability. Pretty much the only market that purchases ordinary SPARC hardware is the legacy customer market.

However, Sun can't just give up on SPARC now, because right now it needs the Niagara SPARCs for its overall 'cool computing' agenda. Sun has to have a power efficient CPU in order to deliver the significant advantages necessary to effectively push forward on cool computing, and the x86 vendors aren't making CPUs that are good enough. Ergo, Sun has to make one itself, and in fact its ability to do so gives it a unique competitive advantage.

The long term problem is that this SPARC advantage is almost certainly dead the moment that Intel decides it needs to seriously chase 'cool computing'. Intel has already buried one competitor that tried to compete on cool CPUs, namely Transmeta, so it can almost certainly do so again if it wants to. So if Sun succeeds in making cool computing a mainline server issue that matters, Intel will have a big motive for chasing the market and will probably squash SPARC's advantages relatively flat. At that point the smart thing to do is to give up on SPARC; trying to compete with Intel is a losing game that Sun has already played once before.

Thus Sun needs to be viewing SPARC not as an end to itself but as a means to an end, a means that Sun is fully prepared to give up on once the end is achieved. Believing otherwise is just asking for Sun's late 90s and early 00s history to repeat itself again.

(This is part of my slow reaction to Tim Bray's What Sun Should Do.)

A thesis: Sun should fork Solaris

Right now, Sun has two markets for Solaris, that being the people who already run Solaris (their current customers), and the people that Sun would like to persuade to run Solaris, based on attractive things like ZFS and DTrace.

The first group has generally been running Solaris for years (usually on SPARC, which is one reason that it lingers on). They care a lot about backwards compatibility, and so they want Solaris administration and so on to keep working just the way they are used to.

The second group has to be romanced away from alternatives like Linux. Backwards compatibility with current Solaris is not an asset; in fact it is a deficit, since while Solaris may be winning on things like DTrace, it is losing horribly on things like modern package and patch management.

The first group, the legacy customers, are important for Sun's current revenue stream. The second group of customers are vital if Sun is to grow (or even to survive; pretty much by definition legacy customers only dwindle over time). Sun can't afford to alienate either group. But they have conflicting requirements for Solaris; the first group wants no changes, while the second group requires Solaris to change, to modernize to match its competitors.

This is an irreconcilable conflict; you cannot have one version of Solaris that is both backwards compatible and modern. Not if you want something that is not a horrible ugly mess internally, with multiple package systems cohabiting uneasily and so on.

So my thesis is that Sun should fork Solaris now. One version would be for the legacy customer base, with compatibility with all of the old tired programs, and one version would be for everyone else, modernized and brought up to competitive speed. The two should have the same base (kernel, core libraries and so on), but the new Solaris could break all sorts of tool-level compatibility in order to deliver something attractive to the customers that can actually grow Sun's business.

I think that such a fork would not need to be much extra work for Sun. Things specific to the legacy environment would presumably be basically static (since change is not considered an advantage there), and Sun already has to maintain the common base. This leaves the new Solaris environment, and Sun has to develop that anyways in order to be competitive. (I am aware that this is somewhat optimistic, but hopefully not too much so.)

(The inspiration for this entry is Tim Bray's What Sun Should Do.)