More on the module selection problem

A commentator on reddit made a decent point on my last entry; to quote:

Sometimes you just have to set aside time to find the best of breed and get to know the code you are using. [...] Exploration has a cost in software projects, and sometimes software projects fail (sub-project in this case).

I agree with this on large, important projects. if you're settling in to make a significant investment of time in programming something, you should expect to take a certain amount of time to figure various things out. If you've got a several week programming project, spending a day to figure out which of three or four modules you want to use is par for the course. Like a lot of other things in programming, this is partly an issue of optimizing costs; it makes sense to spend this time when it's going to make an important difference to your program.

But I also disagree. Not everything we use Python for is large, especially if we can use third party modules to do a lot of the work, and when you're working on small things the balance of time and effort is a lot different.

Much of what I write is relatively small things, and often they're sideline projects as opposed to things that we have to have. With such small things I wind up with three choices:

• I can spend an appreciable amount of time (compared to how much work I expect the rest of the program to be) trying to figure out if the third party module is any good. Even if it is good this can be a mis-optimization because I'm not saving time overall.

• I can slap together my own version of the module. It won't work as well as a good real version, but writing it will probably be faster than trying to find a good real version unless I get lucky.

• I can abandon the program entirely because either of the other options would take more time than can be justified given the program's benefits. There's lots of things that I would like but that I don't have all that much time to spend on.

I often wind up going to the second or third option because the first one can be quite dangerous in a relatively small project. Often the only solid way to evaluate a promising module is to use it for real, ie to try to write my program using it. But if the module turns out not to work, I face the time consuming and seriously demotivating prospect of ripping much of my code apart to take out the bad module and swap in some replacement.

(It's much of my code because in a small thing, outside modules are likely to be fairly significant for the rest of your code instead of contained in a relatively small corner).