An irritating little bug in the latest GNU Emacs Python autoindent code

July 3, 2016

I really like having smart autoindent in my editor when writing code, Python code included. When it works, autoindent does exactly what I would do by hand, does it easier, and in the process shows me errors in my code (if the autoindent is 'wrong', it is a signal that something earlier is off). But the flipside of this is that when autoindent goes wrong it can be a screaming irritation, as I flip from working with my editor to actively fighting it.

Unfortunately the latest official version of GNU Emacs has such an issue in its Python autoindent code, under conditions that are probably rare. To see the bug, set Emacs up with python-indent-offset set to 8 and indent-tabs-mode set to t, and then enter:

def abc():
	if d in e:
	# Hit return here:

If you put your cursor on the end of the comment and hit return, autoindent doesn't add any indentation at all. It should add one level of indentation. Also, once you have this code in a .py file you don't need to set anything in Emacs; Emacs will auto-guess that the indent offset is 8 and then the mere presence of tabs will cause things to explode. This makes this issue especially annoying and/or hazardous.

Some people will say that this serves me right for still using tabs for indentation in my Python code. I'm aware that there's been a general movement in the Python community to indent all Python code with only spaces, regardless of how much you indent it by, but for various reasons I have strongly resisted this. One of them is that I edit Python code in multiple editors, not all of them ones with smart autoindentation, and space-based indenting is painful in an editor that doesn't do it for you. Well, at least using generous indents with manual spaces is painful, and I'm not likely to give that up any time soon.

(I like generous indents in code. Small indent levels make everything feel crammed together and it's less obvious if something is misindented when everything is closer. Of course Python's many levels of nesting doesn't necessarily make this easy; by the time I'm writing an inner function in a method in a class, I'm starting to run out of horizontal space.)

PS: I suspect that I'm going to have to give up my 'indent with tabs' habits some day, probably along with my 8-space indents. The modern Python standard seems to be 4-space indent with spaces and there's a certain amount to be said for the value of uniformity.

(People are apparently working on Python equivalents of Go's gofmt, eg yapf. This doesn't entirely make my issues go away, but at least it would give me some tools to more or less automatically convert existing code over so that I don't have to deal with a mismash of old and new formatting in different files or projects.)

Comments on this page:

You should be feeling right at home with Go then :) , with its recommendation of tab spaced indentation.

By Greg A. Woods at 2016-08-12 14:23:23:

First off I should preface by saying I'm not a frequent Python programmer... and I don't have anything really special for python in my local customization. (I disabled the only thing I think was relevant to test this)

So, which version of GNU Emacs worked as you expected it to work?

I currently use both 23.3 and 24.5 on a daily basis. 24.5 is described as "the current stable release" on the site, so I'm guessing that's the one you refer to as having these issues.

There is no electric-indent-mode in 23.3 (and it also only has python-indent, not python-indent-offset). So if I hit <return> at the end of the "pass" line, the cursor lands in the left-most column.

I also find that the comment line is not correctly indented according to 23.3. If I hit <tab> at the beginning of the comment line it re-indents it to the left-most column. That's odd.

BTW, if I start 23.3 with your test in a file using emacs -q and then start typing a new function definition similar to yours at the end of the file then I have to press <tab> on every line to indent it but it indents with tab characters (indent-tabs-mode is left on in the buffer).

In version 24.5 things work a little more like I think you're expecting them to, and with the bug you describe. Indeed it does not re-indent the comment line to the left-most column. It also auto-indents (to one level, under the "if") if I hit <return> at the end of the "pass" line. However it also fails to in the way you describe by leaving the cursor at the left-most column after hitting <return> at the end of the comment line.

If I start 24.5 with -q and then apply your suggested settings then I cannot force it to indent after the comment line no matter how many times I hit <tab>. However if I start it with my settings I can hit <tab> twice to indent the last line to the level of the comment line. By default of course it has also turned off indent-tabs-mode when I load your test from a file (emacs -q, though as you mention it does guess the local offset for the file correctly as 8.

On the other hand 24.5 with -q and no changes to settings it will allow me to type a new function definition at the into the file, and all lines will auto-indent, including a second comment line, exactly as I think you've described how you want it to work. New indentation is added as space characters though, which I guess I expect, but it's obviously not what is desired when loading a file already indented with tabs.

The problem with 24.5 seems to begin when one turns on indent-tabs-mode. That's when I start to see the screwy behaviour with the comment lines that are indented by tab characters.

Looking in 24.5's python.el it seems the new author of the new python mode really hates indent-tabs-mode.

Probably the problem is in the way the syntax parser works in python.el. You can evaluate (python-indent-context) with the cursor at any point and see what it thinks. If you do so after a comment line indented by tabs, with or without indent-tabs-mode set, it gets it wrong, but if the comment is indented with spaces, it gets it right.

I suppose it might be possible to fix this bug in the new python mode, and possibly even to get it pushed upstream for the next emacs version, though given the situation within the larger Python community as you mention, it may not be worth the effort. If I were a more frequent Python programmer I think I'd still want to stick with real tab character indentation though and so would also want to fix this bug.

I'll mention smart-tabs-mode here for reference, but I'll comment on it more in your more recent post about indentation style....

By cks at 2016-08-12 16:44:48:

The specific versions I've had (and not had) the problem with are Ubuntu 14.04's 24.3 (no problem) and Ubuntu 16.04's 24.5 (with the problem, and it's also there on the current Fedora version of 24.5). We still have some Ubuntu 10.04 machines with 23.3 and on them, in a clean startup ('emacs -q -nw'), I can set-variable python-indent and indent-tabs-mode, use C-j to force indentation on new lines, and have the test indent right. In 23.3, repeatedly hitting tab on a comment line to reindent it seems to toggle it between indented for the function and indented flush left, which is actually potentially useful behavior; after all, what you might want is a new module level comment.

(The behavior is slightly different on 24.5 with 'emacs -q', but all of this is starting to make my head hurt. Some of this undoubtedly comes down to settings in my .emacs. For example, I have a longstanding global key binding for C-m that sets it to newline-and-indent, which maybe should get changed for modern Emacs versions. I sort of restarted my .emacs from scratch due to historical baggage, but not totally from scratch for various reasons, so I still have lingering ancient bits.)

Thanks for the pointer to (python-indent-context) and the data about comments indented with tabs versus spaces. You've clearly found the specific issue/bug, although as you say who knows if the maintainer will accept it (and who knows if a fix would make it back into Ubuntu any time soon).

Written on 03 July 2016.
« cal's unfortunate problem with argument handling
A feature I wish the Linux NFS client had: remapping UIDs and GIDs »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Jul 3 23:02:17 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.