Wandering Thoughts archives

Using argparse in my Python programs encourages me to add options to them

Today, for reasons outside the scope of this entry, I updated one of my very old little Python utilities to add some features and move it to Python 3 in the process. This program was so old it was using getopt, so as part of updating it I switched it over to argparse, which is what I use in all of my modern programs. The change was positive in a variety of ways, but one of the things it did was it immediately caused me to add some more command line options. This isn't due to anything specific to this program, because over and over again I've had much the same experience; my argparse based programs have more options (and often better structured ones).

In thinking about it I believe there's a couple of reasons for this. First, argparse makes it easy to add the basic support for an option in a single place, in a ArgumentParser.add_argument() call. Unlike with getopt's much more primitive argument handling, I don't have to modify code in several places just to have my program accept the option, handle it to set or record some value, and include it in usage and help. Even if the option does nothing this is an easy first step.

Second, that argparse generates an 'args' (or 'options') object with all of the parsed information on it often makes it easy to expose new options to the code that needs to look at them. My usual pattern with argparse is to pass the 'opts' object you get from ArgumentParser.parse_args() down to functions that get called to do the work, rather than break out specific flags, options, and so on separately. This means that a new option is usually pervasively available through the code and doesn't have to be passed in specifically; I can just check 'opts.myNewOption' wherever I want to use it. By contrast, getopt didn't create a natural options object so I tended to pass around options separately (or worse, set global variables because it was the easy lazy way).

This doesn't always work; sometimes I need the new option in a place where I haven't already passed in the opts object. But it works a lot of the time, and when it works it makes it that much easier to code up the use of the new option.

(This also means it's easy to reverse out of an option I decide that I don't want after all. I can delete the creation of it and the use of it without having to go through changing function arguments around. Similarly it's easy to change how an option works because that too doesn't touch many places in the code.)

My first Django database migration worked fine

We have a long standing Django application, using SQLite as the database layer because that's easy and all we need at our tiny scale. For as long as we've had the application I've not so much as breathed on its database schema (which is to say its model), because the thought of trying to do any sort of database migration was a bit scary. For reasons outside the scope of this entry, we recently decided that it was time to add some fields to our application model, so I got to (or had to) try out Django's support for more or less automatic database migrations. The whole experience was pretty painless in my simple case, although I had some learning experiences.

The basics of Django migrations are well covered by Django's documentation. For straightforward changes, you change your model(s) and then run the 'makemigrations' Django subcommand, supplying the name you want for your migration; Django will write a new Python file to do the work. Once you've made the migration you can then apply it with the 'migrate' subcommand, and in at least some cases un-apply it again. Our changes added simple fields that could be empty, which is about the simplest case you can get; I don't know how Django handles more complicated cases, for example introducing a mandatory non-null field.

One learning experience is that Django will want to create a additional new migrations if you tinker with the properties of your (new) model things after you've created (and probably applied) the initial migration in your development environment. For me, this happened relatively naturally as I was writing Django code to use these new fields, now that we had them. You probably don't want to wind up with a whole series of iterating migrations, so you're going to want to somehow squash these down to a single final migration for the production change. Since we use SQLite, I wound up just repeatedly removing the migration file and reverting my development database to start again, rather than tinker around with un-applying the migration and trying to get Django to rewrite it.

(Now that I'm reading the documentation all the way through there's a section on Squashing migrations, but it seems complex and not necessarily quite right for this case.)

While it's not strictly a migration issue as such, one thing that I initially forgot to do when I added new model fields was to also update the settings for our Django admin interface to display them. Partly this happened because it's been so long since I touched this code that I'd somewhat forgotten how it all worked until I looked at the admin interface, didn't see these fields in the particular model object, and started digging.

Although the Django migration documentation contains scary warnings about doing migrations with a SQLite database, I had no problem doing it in production with our generally barely used web application (we normally measure activity in 'hits per month'). Given how Django does database migrations on SQLite, a sufficiently active web application might have to shut down during the migration so that the database could be fiddled with without problems. In general, shutting down will avoid a situation where your running code is incompatible with the state of your database, which can definitely happen even in simple changes.

(Some people probably deploy such a database migration in multiple steps, but with a tiny application we did it the single step way, deploying a new version of the code that used the new fields at the same time as we performed the database migration on the production database.)

My overall experience is quite positive; changing our model was nowhere near as scary or involved as I expected it to be. I suspect I'm going to be much more willing to make model changes in the future, although I still don't think of it as a casual thing.