Wandering Thoughts archives

Some advantages of using argparse to handle arguments as well as options

I started using Python long enough ago that there was only the getopt module, which was okay because that's what I was used to from C and other Unix languages (shell, Perl, etc), and then evolved for a bit through optparse; I only started using argparse relatively recently. As a result of all of this background, I'm used to thinking of 'argument handling' as only processing command line switches and their arguments for you, and giving you back basically a list of the remaining arguments, which is your responsibility to check how many there are, parse, and so on.

Despite being very accustomed to working this way, I'm starting to abandon it when using argparse. Part of this is what I discovered the first time I used argparse, namely that it's the lazy way. But I've now used argparse a second time and I'm feeling that there are real advantages to letting it handle as many positional arguments as possible in as specific a way as possible.

For instance, suppose that you're writing a Python program that takes exactly five positional arguments. The lazy way to handle this is simply:

parser.add_argument("args", metavar="ARGS", nargs=5)

If you take exactly five arguments, they probably mean different things. So the better way is to add them separately:

parser.add_argument("eximid", metavar="EXIMID")
parser.add_argument("ctype", metavar="CONTENT-TYPE")
parser.add_argument("cdisp", metavar="CONTENT-DISPOSITION")
parser.add_argument("mname", metavar="MIMENAME")
parser.add_argument("file", metavar="FILE")

Obviously this gives you easy separate access to each argument in your program, but the really nice thing this does is that it adds some useful descriptive context to your program's usage message. If you choose the metavar values well, your usage message will strongly hint to what needs to be supplied as each argument. But we can do better, because argparse is perfectly happy to let you attach help to positional arguments as well as to switches (and it will then print it out again in the usage message, all nicely formatted and so on).

You can do the same thing by hand, of course; there's nothing preventing you from writing the same documentation with manual argument parsing and printing it out appropriately (although argparse does do a good job of formatting it). But it feels easier with argparse and it feels more natural, because argparse lets me put everything to do with a positional argument in one spot; I can name the internal variable, specify its visible short name, and then add help, all at once. If nothing else, this is likely to keep all of these things in sync with each other.

(And I'm not going to underestimate the importance of automatic good formatting, because that removes a point of friction in writing the help message for a given positional argument.)

The result of all of this is that using argparse for positional arguments in my latest program has effortlessly given me not just a check for having the right number of positional arguments but a bunch of useful help text as well. Since I frequently don't touch programs for a year or two, I foresee this being a useful memory jog for future me.

In summary, if I can get argparse to handle my positional arguments in future Python programs, I'm going to let it. I've become convinced that it's not just the lazy way, it's the better way.

(This is where some Python people may laugh at me for having taken so long to start using argparse. In my vague defense, we still have some machines without Python 2.7.)

Today's lesson on the value of commenting your configuration settings

We have a relatively long-standing Django web application that was first written for Django 1.2 and hadn't been substantially revised since. Earlier this year I did a major rework in order to fully update it for Django 1.9; not just to be compatible, but to be (re)structured into the way that Django now wants apps to be set up.

Part of Django's app structure is a settings.py file that contains, well, all sorts of configuration settings for your system; you normally get your initial version of this by having Django create it for you. What Django wants you to have in this file and how it's been structured has varied over Django versions, so if you have a five year old app its settings.py file can look nothing like what Django would now create. Since I was doing a drastic restructuring anyways, I decided to deal with this issue the simple way. I'd have Django write out a new stock settings.py file for me, as if I was starting a project from scratch, and then I would recreate all of the settings changes we needed. In the process I would drop any settings that were now quietly obsolete and unnecessary.

(Since the settings file is just ordinary Python, it's easy to wind up setting 'configuration options' that no longer exist. Nothing complains that you have some extra variables defined, and in fact you're perfectly free to define your own settings that are used only by your app, so Django can't even tell.)

In the process of this, I managed to drop (well, omit copying) the ADMINS setting that makes Django send us email if there's an uncaught exception (see Django's error reporting documentation). I didn't spot this when we deployed the new updated version of the application (I'm not sure I even remembered this feature). I only discovered the omission when Derek's question here sent me looking at our configuration file to find out just what we'd set and, well, I discovered that our current version didn't have anything. Oops, as they say.

Looking back at our old settings.py, I'm pretty certain that I omitted ADMINS simply because it didn't have any comments around it to tell me what it did or why it was there. Without a comment, it looked like something that old versions of Django set up but new versions didn't need (and so didn't put into their stock settings.py). Clearly if I'd checked what if anything ADMINS meant in Django 1.9 I'd have spotted my error, but, well, people take shortcuts, myself included.

(Django does have global documentation for settings, but there is no global alphabetical index of settings so you can easily see what is and isn't a Django setting. Nor are settings grouped into namespaces to make it clear what they theoretically affect.)

This is yet another instance of some context being obvious to me at the time I did something but it being very inobvious to me much later. I'm sure that when I put the ADMINS setting into the initial settings.py I knew exactly what I was doing and why, and it was so obvious I didn't think it needed anything. Well, it's five years later and all of that detail fell out of my brain and here I am, re-learning this lesson yet again.

(When I put the ADMINS setting and related bits and pieces back into our settings.py, you can bet that I added a hopefully clear comment too. Technically it's not in settings.py, but that's a topic for another blog entry.)

What Python 3 versions Django supports, and when this changes

I was idly skimming the in-progress release notes for Django 1.10 when one of the small sections that I usually ignore jumped out at me instead:

Like Django 1.9, Django 1.10 requires Python 2.7, 3.4, or 3.5. [...]

Since I've recently been thinking about running Django on Python 3, the supported Python 3 versions caught my eye. More exactly, that it was a short list. This made me wonder what Django versions will support what Python 3 versions, and for how long.

In mid 2015, the Django project published a roadmap and said:

We will support a Python version up to and including the first Django LTS release whose security support ends after security support for that version of Python ends. For example, Python 3.3 security support ends September 2017 and Django 1.8 LTS security support ends April 2018. Therefore Django 1.8 is the last version to support Python 3.3.

So we need to look at both the Django release schedule and the Python release and support schedule. On the Django side, Django's next LTS release is '1.11 LTS', scheduled to release in April 2017 and be supported through April 2020 (and it's expected to be the last version supporting Python 2.7, since official Python 2.7 security support ends in 2020). After that is Django 2.2 in April 2019, supported through April 2022. On the Python side, the Python team appears to be doing 3.x releases roughly every 18 months (see eg PEP 494 on Python 3.6's release schedule) and giving them security support for five years after their initial release. If this is right, Python 3.4 will be supported through March 2019 and Python 3.5 through September 2020; 3.6 is expected in December 2016 (supported through December 2021) and thus 3.7 in roughly May of 2018 (supported through May 2023).

Putting all of this together, I get an expected result of:

• Python 3.4 will be supported through Django 1.11; Django 2.0 (nominally due December 2017) will drop support for it.

• Python 3.5 and 3.6 will probably be supported through Django 2.2. Django 1.11 will almost certainly be the first release to support Python 3.6.

• Python 3.7's exact Django support range is up in the air since at this point I'm projecting both Python and Django release schedules rather far into the misty future.

Ubuntu 14.04 LTS has Python 3.4 and Ubuntu 16.04 LTS has 3.5. Both will be supported long enough to run into the maximum likely Django version that supports their Python 3 version, although only more or less at the end of each Ubuntu LTS's lifespan.

(I'm going to have to mull over what this means for Python 3 migration plans for our Django app. Probably a real Python 3 migration attempt is closer than I thought it would be.)

I need to cultivate some new coding habits for Python 3 ready code

We have a Django application, and because of various aspects of Django (such as Django handling much of the heavy lifting), I expect that it's our most Python 3 ready chunk of code. Since I was working on it today anyways, I took a brief run at seeing if it would at least print help messages if I ran it under Python 3. As it turns out, making the attempt shows me that I need to cultivate some new coding habits in order to routinely write code that will be ready for Python 3.

What I stumbled over today is that I still like to write except clauses in the old way:

try:
   ....
except SomeErr, err:
   ....

The new way to write except clauses is the less ambiguous 'except SomeErr as err:'. Python 3 only supports the new style.

Despite writing at least some of the code in our Django application relatively recently, I still wrote it using the old style for except. Of course this means I need to change it all. I'm pretty certain that writing except clauses this way is not something that I think about; it's just a habit of how I write Python, developed from years of writing Python before 'except CLS as ERR' existed or at least was usable by me.

What I take away from this is that I'm going to need to make new Python coding habits, or more exactly go through the more difficult exercise of overwriting old habits with new ones. I'm sure to be irrationally annoyed at some of the necessary changes, especially turning 'print' statements into function calls.

(If I was serious about this, what I should do is force myself to write only in Python 3 for a while. Unfortunately that's not very likely.)

The good news is that I checked some code I wrote recently, and I seem to have deliberately used the new style except clauses in it. Now if I can remember to keep doing that, I might be in okay shape.

(Having thought about it, what would be handy is a Python linter that complains about such unnecessary Python 3 incompatibilities. Then I'd at least have a chance of catching my issues here right away.)

PS: Modernizing an old code base is a related issue, of course. I need to modernize both code and habits to be ready for Python 3 in both current and future code.

Sidebar: The timing and rationality of using old-style except

New style except was introduced in Python 2.6, which dates back to 2008. However, the new version of Python didn't propagate into things like Linux distributions immediately; it took two years to get it into an Ubuntu LTS release, for example (in 10.04). Looking back at various records, it seems that the initial version of our Django application was deployed on an Ubuntu 8.04 machine that would have had only Python 2.5. In fact I may have written the first version of all of the substantial code in the application while we were still using 8.04 as the host machine and so new-style except would have been unavailable to us.

This is of course no longer the case. Although not everything we run today has Python 2.7 available (cf), it all has at least Python 2.6. So I should be writing all my code with new style except clauses and probably some other modernizations.

Moving from Python 2 to Python 3 calls for a code inventory

I was all set to write a blog entry breaking down what sort of Python code we had, how much it was exposed to security issues and other threats, and how much work it would probably be to migrate it to or towards Python 3. I even thought it was going to be a relatively short and simple entry. Then, as I was writing things down, I kept remembering more and more bits of Python code we're using in different contexts, and I'm pretty sure I'm still forgetting some.

So, here's my broad moral for today: if you have Python code, and you're either thinking of migrating at least some of it to Python 3 or considering whether you can ignore the alleged risks of continuing to use Python 2, your first step is (or should be) to get a code inventory. Expect this to probably take a while; you don't want just the big obvious applications, you also care about the little things in the corners.

Perhaps we're unusual, but we don't have our Python code in one or more big applications, where it's easy and obvious to look at things. Instead, we have all sorts of things written in Python, everything from a modest Django application through system management subsystems to little command line things (and not so little ones). These have accumulated over what is almost a decade by now, and if they work quietly we basically forget about them (and most of them do). It's clearly going to take me some work to find them all, categorize them, and probably in some cases discover that they're now unnecessary.

Having written this, I don't know if I'm actually going to do such an inventory any time soon. The problem is that the work is a boring slog and the issue is not particularly urgent, even if we accept a '2020 or earlier' deadline on Python 2 support. Worse, if I do an inventory now and then do nothing with it, it's probably going to get out of date (wasting the work). I'd still like to know, though, if only for my own peace of mind.