Some ways to add versioning to pickled objects

April 20, 2009

To follow on from yesterday's entry, suppose that one is using (c)Pickle to save data in your Python program, and you want to version your data somehow. I can think of a number of approaches:

  • take the 'pickle as stored JSON' approach; serialize your complex objects to simple objects (dictionaries, lists, etc), add version numbers, and only pickle the simple objects. Then you can do all of the usual version mismatch fixups when you de-serialize the reloaded simple objects back to your complex objects.

  • version your class names; instead of having, say, a Comment class and pickling several different versions of it, have a CommentV1 class, a CommentV2 class, and so on. (I imagine that you will want a Comment abstract class to have all of the common behavior.)

  • don't explicitly version things. Take advantage of the fact that pickle doesn't actually initialize objects as such, just stuffs data into their __dict__, and write your object methods such that they will deal with any set of data they could get from any version of your objects. (Renaming instance variables may help.)

    The easiest approach to this is probably to call a fixup method on newly loaded objects; this method can then canonicalize old data versions into the current world.

  • define a custom __setstate__ method that works out what version of the data that it's restoring based on the contents of the dictionary that it's handed. This is essentially the fixup method approach, just automated, and you have to copy the data onto the object yourself.

All of these have drawbacks, and some of them are ugly. If I had to do any of these I would probably take the 'pickle as stored JSON' approach; although it is one of the more annoying choices (since you write a bunch of code), it is the least ugly.

(The custom __setstate__ approach has a pleasing minimalism but involves a little bit too much magic to make me happy.)

Written on 20 April 2009.
« Why pickle is not a good way to save your data
Why your ticketing system should not be accessible to users »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 20 23:46:24 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.