python/MarshalVsCPickle written at 23:19:41; Add Comment
The Python marshal module versus the cPickle module
The marshal module looks interesting for persisting and retrieving lightweight data, but the big question to me has always been whether in exchange for constraining your data down to simple structures of primitive types you got something that was actually faster than the cPickle module.
So today I decided to finally answer the question by doing some timing tests. I won't claim that these are comprehensive or entirely scientific, but I do have some results:
Since DWiki's cache layer spends a lot of time writing and reading long strings, it looks like I made the right decision way back when. The combination of long strings and floating point numbers meant that marshal was significantly slower than cPickle for a simulated DWiki cache object.
The general parity in loading time suggests that even for simple data structures, for a caching layer you might as well use cPickle; you are not particularly slower for the thing you're going to be doing a lot, and you get a bunch of (potential) benefits in return.
* * *
Atom feeds are available; see the bottom of most pages.