Wandering Thoughts archives

It's easier to shrink RAID disk volumes than to reshape them

Once your storage system is using more than a single disk to create a pool of storage, there are a number of operations that you can want to do in order to restructure that pool of storage. Two of them are shrinking and reshaping. It's common for volume managers and modern filesystems like btrfs to be able to shrink storage pool by removing a disk (or a set of mirrored disks), although not all modern filesystems support doing this. It's also becoming increasingly common for RAID (sub)systems to support reshaping RAID pools to do things like change from RAID-5 to RAID-6 (or vice versa); modern filesystems may also implement this sort of reshaping if they support RAID levels that can use it. Often shrinking and reshaping are lumped together as 'yeah, we support reorganizing storage in general'.

In thinking about this whole area lately, I've realized that shrinking is fundamentally easier to do than reshaping because of what it involves at a mechanical level. When you shrink a pool of storage, you do so by moving data to a new place; you move it from disk A, which you are getting rid of, to free space on other disks. When all the data has been moved off of disk A, you're done. By contrast, reshaping is almost always an in-place operation. You don't copy all the data to an entirely different set of disks, then copy it back in a different arrangement; instead you must very carefully shuffle it around in place, keeping exacting records of what has and hasn't been shuffled so you know how to refer to it.

For obvious reasons, filesystems et al already have plenty of code for allocating, writing, and freeing blocks. To implement shrinking, 'all' you need is an allocation policy that says 'never allocate on this entity' plus something that walks over the entire storage tree, finds anything allocated on the to-be-removed disk, triggers a re-allocation and re-write, and then updates bits of the tree appropriately. The tree walker is not trivial, but because all of this mimics what the filesystem is already doing you have natural answers for many questions about things like concurrent access by ordinary system activity, handling crashes and interruptions, and so on. Fundamentally, the whole thing is always in a normal and consistent state; it just has less and less of your data on the to-be-removed disk over time.

This is not true for reshaping. Very few storage systems do anything like a RAID reshaping during normal steady state operation. This means you need a whole new set of code, you're going to have to be very careful to manage things like crash resistance, and a pool of storage that's in the middle of a reshaping looks very different from how it does in normal operation (which means that you can't just abandon a reshaping in mid-progress in the way you can abandon a shrink).

(This is a pretty obvious thing if you think about it, but I hadn't really considered it before now.)

PS: Not all 'shrinking' is actually shrinking in the form I'm considering here. Removing one disk from a RAID-5 or RAID-6 pool of storage is really a RAID reshape.

(It's theoretically possible to design a modern filesystem where RAID reshapes proceed like shrinking. I don't think anyone has done so, although maybe this is how btrfs works.)

My views of Windows 10 (from the outside)

This sort of starts with my tweet:

It's funny; I have a great deal of anger for what Microsoft has done with Windows 10, even though none of it affects me directly.

Since I was asked, I'm going to change my mind and write enough here to explain myself.

Based on information casually available to an outsider to the Windows ecosystem, Microsoft has done two things with Windows 10. First, they have significantly to drastically increased the amount of privacy invasive 'telemetry' that Windows 10 installs send to Microsoft and have also added all sorts of advertising to it. The normal versions of Windows 10 will pitch you in the Start menu, on lock screens, in included Windows applications like Solitaire, and so on.

Second, as everyone has heard by now, Microsoft has been aggressively pushing the upgrade to Windows 10 on people (or more accurately Windows machines). At this point it seems to be almost impossible to escape the upgrade; certainly it requires so many contortions that many people will be upgraded even if they don't want to be. Stories abound about important PCs in various places basically being hijacked by these forced upgrades.

All by themselves, either of these things would be bad and obnoxious; no one wants ads, invasive telemetry, or forced upgrades. Together they ascend to an entirely new level of nastiness, as Microsoft is forcing you to upgrade to an intrusive, ad-laden new operating system (and they've made it clear that the amount of ads will be increasing over time). The whole thing also comes at what could politely be called a bad time for both ads and privacy intrusion; people are becoming more and more sensitized and angry about both, as we see with the popularity of adblockers and so on.

In my view, what Microsoft has done is to reveal that as long as you use a Microsoft operating system, your computer really belongs to Microsoft instead of you. By forcing this upgrade to an OS with very different behavior for advertising and privacy intrusion, Microsoft has now demonstrated that they are willing to drastically change the terms on which they let you use your computer, as they see fit. Your computer and OS does not exist to benefit you, it exists to benefit Microsoft. If it is not doing enough for them, they will change things until it does and you do not get a vote in the matter.

(Microsoft could try to sell more telemetry as better for you, but that is absolutely impossible with ads. Ads universally make your experience worse. By including and then increasing ads in Windows 10, Microsoft is clearly prioritizing themselves over you in the operating system.)

In my view, by doing this Microsoft has shown that they are not particularly different from the big OEMs who have for years been loading down Windows laptops and desktops with pre-installed crapware. Dell, HP, Lenovo, et al have all been more than willing to ruin the experience for people buying their hardware in order to make some additional money from other channels; now Microsoft has joined the crowd. As a result, Microsoft is just as un-trustworthy as the big OEMs are.

(More fundamentally, Microsoft is showing that they do not care about people's experience of using their operating systems, or at least that they don't consider it a priority. Microsoft will happily make your time using Windows 10 less pleasant in order to deliver some ads. And as you know, when you are clearly not the customer, you are the product. It is especially offensive to be the product when you are paying for the privilege, but apparently that is life in Microsoft's world.)

I very much hope that this winds up causing Microsoft massive problems down the road. There certainly should be consequences to changing your product from a premium thing that was the best solution to a downmarket option used by people who don't have the money to avoid the annoyances it inflicts on them. However, I cynically doubt that it will, and it may be that Microsoft Windows has already become the downmarket product that Windows 10 positions it as.

In the mean time the whole situation makes me angry every time I consider it, especially when I think of the various relatives and people I know who will have no choice but to use Windows 10 and be subjected to all of this. If Microsoft goes down in flames someday, this move of theirs has made sure that I will applaud the fires.

Sidebar: The danger of intrusive telemetry

The ever more intrusive (default) telemetry makes me especially angry because if there is one thing we have learned over the past three, five, or ten years, it is that collecting and retaining data is inherently dangerous. Once that data exists it becomes a magnet for people who want a look at it, whether that is with subpoenas in civil lawsuits, warrants in criminal cases, or NSLs from three letter agencies. Today, the only safe thing to do with data is not collect it at all or at the very least, totally minimize your collection. That Microsoft has chosen to do otherwise basically amounts to them shrugging their shoulders over the fundamental privacy of people using their operating system.

(Now we know how much Microsoft really cares about the privacy of people using their systems, as opposed to things that cause inconvenience or bad PR for Microsoft.)