What causes the ZFS file deletion problem with snapshots

April 16, 2009

Suppose that you have a ZFS pool that you have filled utterly up to the brim, completely full, and that it has a snapshot. Of course, now you need to delete some files in a filesystem to clean up space, so:

$ rm oldfile
rm: cannot remove `oldfile': No space left on device

(Here oldfile is a file that is old enough to be in the snapshot.)

The ultimate cause of this is a general problem with 'copy on write' filesystems such as ZFS. In these filesystems, you never overwrite anything, either data or filesystem metadata; any time you change anything, you have to write it to a new location (and then update all of the things pointing to it, all the way up to the root (which technically does get overwritten, sort of)).

This applies to file deletions just as much as it applies to anything else, as deleting a file means updating the directory it is (or was) in and so on. In theory, when there is no free space in the pool this means that there is no new location to put that update, and so removing a file can fail. In practice copy on write filesystems, ZFS included, are smart enough to know that this is stupid, so they normally manage to find some temporary extra space for this situation.

What's happened in our example is that snapshots have complicated the picture. Deleting a file that is also in a snapshot doesn't actually free up any space (since the file has to stick around for the snapshot). Instead, it means that you really do need new space for a new copy of all of the relevant metadata (to mark the file as deleted in the current version), and when the ZFS pool is full, there is no room for this new copy and so a removal has to fail.

Written on 16 April 2009.
« The problem with Solaris 10 update 6's ZFS failmode setting
Git and 'rewriting history' »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 16 01:16:46 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.