Wandering Thoughts archives

ZFS on Linux's development version now has much better pool recovery for damaged pools

Back in March, I wrote about how much better ZFS pool recovery was coming, along with what turned out to be some additional exciting features, such as the long-awaited feature of shrinking ZFS pools by removing vdevs. The good news for people using ZFS on Linux is that most of both features have very recently made it into the ZFS on Linux development source tree. This is especially relevant and important if you have a damaged ZFS on Linux pool that either doesn't import or panics your system when you do import it.

(These changes are OpenZFS 9075 and its dependencies such as OpenZFS 8961, and the vdev removal changes, although there are followup fixes to them such as OpenZFS 9290.)

These changes aren't yet in any ZFS on Linux release and I suspect that they won't appear until 0.8.0 is released someday (ie, they won't be ported into the current 0.7.x release branch). However, it's fairly easy to build ZFS on Linux from source if you need to temporarily run the latest version in order to recover or copy data out of a damaged pool that you can't otherwise get at. I believe that some pool recovery can be done as a one-time import and then you can revert back to a released version of ZFS on Linux to use the now-recovered pool, but certainly not all pool import problems can be repaired like this.

(As far as vdev removal goes, it currently requires permanently using a version of ZFS that supports it, because it adds a device_removal feature to your pool that will never deactivate, per zpool-features. This may change at some point in the future, but I wouldn't hold my breath. It seems miraculous enough that we've gotten vdev removal after all of these years, even if it's only for single devices and mirror vdevs.)

I haven't tried out either of these features, but I am running a recently built development version of ZFS on Linux with them included and nothing has exploded so far. As far as things go in general, ZFS on Linux has a fairly large test suite and these changes added tests along with their code. And of course they've been tested upstream and OmniOS CE had enough confidence in them to incorporate them.

Sorting out some of my current views on operator overloading in general

Operator overloading is a somewhat controversial topic in programming language design and programming language comparisons. To somewhat stereotype both sides, one side thinks that it's too often abused to create sharp-edged surprises where familiar operators do completely surprising things (such as << in C++ IO). The other side thinks that it's a tool that can be used to create powerful advantages when done well, and that its potential abuses shouldn't cause us to throw it out entirely.

In general, I think that operator overloading can be used for at least three things:

1. implementing the familiar arithmetic operations on additional types of numbers or very strongly number-like things, where the new implementations respect the traditional arithmetic properties of the operators; for example + and * are commutative.

2. implementing these operations on things which already use these operators in their written notation, even if how the operators are used doesn't (fully) preserve their usual principles. Matrix multiplication is not commutative, for example, but I don't think many people would argue against using * for it in a programming language.

3. using these operators simply for convenient, compact notation in ways that have nothing to do with arithmetic, mathematical notation, or their customary uses in written form for the type of thing you're dealing with.

I don't think anyone disagrees with the use of operator overloading for the first case. I suspect that there is some but not much disagreement over the second case. It's the third case that I think people are likely to be strongly divided over, because it's by far the most confusing one. As an outside reader of the code, even once you know the types of objects involved, you don't know anything about what's actually happening; you have to read the definition of what that type does with that operator. This is the 'say what?' moment of << in C++ IO and % with Python strings.

Languages are partly a cultural thing, not purely a technical one, and operator overloading (in its various sorts) can be a better or a worse fit for different languages. Operator overloading probably would clash badly with Go's culture, for example, even if you could find a good way to add it to the language (and I'm not sure you could without transforming Go into something relatively different).

(Designing operator overloading into your language pushes its culture in one direction but doesn't necessarily dictate where you wind up in the end. And there are design decisions that you can make here that will influence the culture, for example requiring people to define all of the arithmetic operators if they define any of them.)

Since I'm a strong believer in both the pragmatic effects and aesthetic power of syntax, I believe that even operator overloading purely to create convenient notation for something can be a good use of operator overloading in the right circumstances and given the right language culture. Generally the right circumstances are going to be when the operator you're overloading has some link to what the operation is doing. I admit that I'm biased here, because I've used the third sort of operator overloading from time to time in Python and I think it made my code easier to read, at least for me (and it certainly made it more compact).

(For example, I once implemented '-' for objects that were collections of statistics, most (but not all) of them time-dependent. Subtracting one object from another gave you an object that had the delta from one to the other, which I then processed to print per-interval statistics.)

In thinking about this now, one thing that strikes me is that an advantage of operators over function calls is that operators tend to be written with whitespace, whereas function calls often run everything together in a hard to read blur. We know that whitespace helps readability, so if we're going to lean heavily on function calls in a language (including in the form of method calls), perhaps we should explore ways of adding whitespace to them. But I'm not sure whitespace alone is quite enough, since operators are also distinct from letters.

(I believe this is where a number of functional languages poke their heads up.)