On the surface, the (c)Pickle module looks like a good, simple way for your Python program to save and load its data; much like XML, it means you don't have to write a parser or even save and load routines as such, just some file and object manipulation code. However, through my experience in writing DWiki I've come to understand that this temptation would be a mistake (one that I've actually half-made; DWiki's caching layer uses pickling).

Fundamentally the problems with pickle for saving data are inherent in what it exists to do; it exists to persist and recover Python objects, not save and restore data. These sound similar enough on first look, but in the longer term I think you run into some significant issues:

• pickle has no concept of versioning for your data structures, which makes it hard to change the data that you store for a particular sort of thing. If you need this (and you will), you will have to resort to various workarounds to build it yourself.

  (In fact pickle doesn't even notice if there is a mismatch between what instance data was pickled for an object and what the object should now have.)

• your data files are not easily inspectable. Yes, I know, pickle has an ASCII version of its storage protocol, but this is still not very readable by hand, and I don't think it's modifiable by hand at all (well, not practically). Essentially pickled things are opaque; the only way to deal with them sensibly is through pickle itself.

• I don't think that pickle has any real concept of error recovery, and with it any way to get partial information for a partially complete data structure. You either get the whole object (or object hierarchy) or you get nothing.

This is not to say that pickle is pointless. It's just that if you're using it, you need to be sure that you really do want objects, not just data.

If you still want to use pickle as your save format because it's easy, I've come around to the idea that you should not attempt to pickle your objects directly. Instead I think that you should treat pickle like you would JSON, and first serialize your actual objects into simple data structures (dictionaries, lists, etc) and pickle only the data structures.

(Admittedly, this is easy for me to say because my use of pickle to date has been for objects that are relatively easily represented this way.)