Why pickle is not a good way to save your data

April 20, 2009

On the surface, the (c)Pickle module looks like a good, simple way for your Python program to save and load its data; much like XML, it means you don't have to write a parser or even save and load routines as such, just some file and object manipulation code. However, through my experience in writing DWiki I've come to understand that this temptation would be a mistake (one that I've actually half-made; DWiki's caching layer uses pickling).

Fundamentally the problems with pickle for saving data are inherent in what it exists to do; it exists to persist and recover Python objects, not save and restore data. These sound similar enough on first look, but in the longer term I think you run into some significant issues:

  • pickle has no concept of versioning for your data structures, which makes it hard to change the data that you store for a particular sort of thing. If you need this (and you will), you will have to resort to various workarounds to build it yourself.

    (In fact pickle doesn't even notice if there is a mismatch between what instance data was pickled for an object and what the object should now have.)

  • your data files are not easily inspectable. Yes, I know, pickle has an ASCII version of its storage protocol, but this is still not very readable by hand, and I don't think it's modifiable by hand at all (well, not practically). Essentially pickled things are opaque; the only way to deal with them sensibly is through pickle itself.

  • I don't think that pickle has any real concept of error recovery, and with it any way to get partial information for a partially complete data structure. You either get the whole object (or object hierarchy) or you get nothing.

This is not to say that pickle is pointless. It's just that if you're using it, you need to be sure that you really do want objects, not just data.

If you still want to use pickle as your save format because it's easy, I've come around to the idea that you should not attempt to pickle your objects directly. Instead I think that you should treat pickle like you would JSON, and first serialize your actual objects into simple data structures (dictionaries, lists, etc) and pickle only the data structures.

(Admittedly, this is easy for me to say because my use of pickle to date has been for objects that are relatively easily represented this way.)


Written on 20 April 2009.
« Sometimes you don't want behavior with your data
Some ways to add versioning to pickled objects »

These are my WanderingThoughts
(About the blog)

Full index of entries
Recent comments

This is part of CSpace, and is written by ChrisSiebenmann.
Twitter: @thatcks

* * *

Categories: links, linux, programming, python, snark, solaris, spam, sysadmin, tech, unix, web

This is a DWiki.
GettingAround
(Help)

Search:

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Apr 20 01:46:52 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.