Chris's Wiki :: blog/python/WhyInStandardLibrary

Why I like modules to be in the Python standard library

September 15, 2010

Even when there may be perfectly good third party modules for something, I really want there to be a module for it in Python's standard library. Part of the reason is obviously how I find third party modules to be awkward, but another part of it is what I call the selection problem.

The selection problem is the problem of picking a third party module (sometimes even finding it, although pypi helps with that) and figuring out if it's any good. Simply figuring out the quality of a module is a bunch of work, and the amount of work multiplies drastically if there's several third party modules that all do what I want. Often, the only way I can really tell if a module is going to work well is to actually try using it. Generally this has to be in a real program (I find toy examples both frustrating to write and uninformative), which means that if I have picked poorly I may have wasted a bunch of time and effort. Even if I can rule out a module relatively early, I had to spend the time to read documentation or skim code or the like, and that time's all wasted.

(And frankly it's frustrating to run into near misses, modules that almost do what I need and almost work. Faced with this, it often at least feels easier to write something from scratch myself if what I want isn't too big.)

When a module has made it into the standard library, I don't have to go through all of this; I can just use the module, secure in the confidence that this is a good implementation of whatever it is that I want to do. Someone else has already gone through all of this quality assurance work, and if there were multiple implementations the Python people have probably either picked the best one or at least determined that they are more or less equivalent and so I am not missing anything very important by not looking at the other options.

(Yes, sometimes this confidence is misplaced. But generally it's at least close.)

Update: see also WhyInStandardLibraryII for additional comments on the time drain of the selection problem.

Comments on this page:

From 211.189.163.250 at 2010-09-16 21:53:51:

Perl + CPAN rocks.

By cks at 2010-09-17 00:52:24:

My view is that Perl + CPAN has the selection problem in spades on top of spades because it has so many packages and thus so many options for pretty much anything you want to build. For example, how many class builder CPAN packages are there, and how are people who not experts supposed to evaluate which one is the right answer in any short amount of time?

From 203.17.251.71 at 2010-09-17 04:48:41:

Plus, Perl+CPAN has the bigger problem that there are some core modules that are incompatible. What do you do if you need to use two modules, one of which requires Moose, and another which requires something that doesn't work with Moose? (Apologies for the in unspecificity, but I can't remember/find what said other module is).

James

From 150.101.221.162 at 2010-09-17 04:55:55:

+1

I just hit the fact that there seems to be no way to encrypt a stream in python because there is no crypto in the core library (except for ssl which seems like overkill). ;-(

From 77.249.14.105 at 2010-09-22 04:17:20:

I quite like the idea of "batteries included" from Python, but must agree that Perl + CPAN is at this point a much better choice if you need something not in the standard library. Besides, if one of the CPAN modules becomes so used it finally enters the core distribution of Perl.

I have been reviewing lately a nagios check written in python (it checks a website and parses the html output) and it is a bloody mess of regular expressions. The script simply screams to use HTML::TableExtract which comes packaged in most distributions. I have rewritten it in Perl and it is now much shorter and easier to understand than the original one because of that module. The reason I had to take a look at it was that the web check changed its web layout, so the regex was not working. Now with HTML::TableExtract I do not have to reinvent the regex everytime this happens, I can simply use the functionality of the module to parse the different fields of a table in HTML. Much cleaner and not error prone.

Anyway, the Perl/Python standard library is just what you as admin and your config management tool make of it ;-)

natxo

From 77.249.14.105 at 2010-09-22 04:21:40:

And about the problem of the huge amount of modules and how to know which one to choose: if you don't know something, you can just ask the experts on irc or on a mailing list, I do not see that as a problem at all. You get the best live support money cannot buy.

natxo

Written on 15 September 2010.

«	An overview of the Debian and RPM source package formats