Wandering Thoughts archives

Adding a new template filter in Django 1.9, and a template tag annoyance

As the result of my discovery about Django's timesince introducing nonbreaking spaces, I wanted to fix this. Fixing this requires coding up a new template filter and then wiring it into Django, which took me a little bit of flailing around. I specifically picked Django 1.9 as my target, because 1.9 supports making your new template filters and tags available by default without a '{% load ... %}' statement and this matters to us.

When you are load'ing new template widgets, your files have to go in a specific and somewhat annoying place in your Django app. Since I wasn't doing this, I was free to shove my code into a normal .py file. My minimal filter is:

from django import template
from django.template.defaultfilters import stringfilter

register = template.Library()

@register.filter
@stringfilter
def denonbreak(value):
   """Replace non-breaking spaces with plain spaces."""
   return value.replace(u"\xa0", u" ")

The resulting filter is called denonbreak. Although the documentation doesn't say so explicitly, you are specifically handed a Unicode string and so interacting with it using plain strings may not be reliable (or work at all). I suppose this is not surprising (and people using Python 3 expect that anyways).

To add your filter(s) and tag(s) as builtins, you make use of a new Django 1.9 feature in the normal template backend when setting things up in settings.py. This is easiest to show:

TEMPLATES = [
  {
    'BACKEND': 'django.template.backends.django.DjangoTemplates',
    [...]
    'OPTIONS': {
       'builtins': ['accounts.tmplfilters'],
       [...]

(Do not get diverted to 'libraries'; it is for something else.)

At this point you might ask why I care about not needing to {% load %} my filter. The answer is one of the features of Django templates, which is that there is no good way to suppress newlines at the end of template directives.

Suppose you have a template where you want to use your new tag:

{% load something %}
The following pending account requests haven't been
handled for at least {{cutoff|timesince|denonbreak}}:
[...]

Django will remove the {% load %}, but it won't remove the newline after it. Thus your rendered template will wind up starting with a blank line. In HTML this is no problem; surplus blank lines silently disappear when the browser renders the page. But in plain text it's another story, because now that newline is sticking around, clearly visible and often ugly. To fix it you must stick the {% load %} at the start of the first real line of text, which looks ugly in the actual template.

({% if %} is another template tag that will bite you in plaintext because of this. Basically any structuring tag will. I really wish Django had an option to suppress the trailing newline in these cases, but as far as I know it doesn't.)

This issue is why I was willing to jump to Django 1.9 and use the 'builtins' feature, despite what everyone generally says about making custom things be builtins. I just hate what happens to plaintext templates otherwise. Ours are ugly enough as it is because of other tags with this issue.

Django, the timesince template filter, and non-breaking spaces

Our Django application uses Django's templating system for more than just generating HTML pages. One of the extra things is generating the text of some plaintext email messages. This trundled along for years, and then a Django version or two ago I noticed that some of those plaintext emails had started showing up not as plain ASCII but as quoted-printable with some embedded characters that did not cut and paste well.

(One reason I noticed is that I sometimes scan through my incoming email with plain less.)

Here's an abstracted version of such an email message, with the odd bits italicized:

The following pending account request has not been handled for at least 1 week.

• <LOGIN> for Some Person <user@somewhere>
  Sponsor: A professor
  Unhandled for 1 week, 2 days (since <date>)

In quoted-printable form the spaces in the italicized bits were =C2=A0 (well, most of them).

I will skip to the punchline: these durations were produced by the timesince template filter, and the =C2=A0 is the utf-8 representation of a nonbreaking space, U+00A0. Since either 1.5 or 1.6, the timesince filter and a couple of others now use nonbreaking spaces after numbers. This change was introduced in Django issue #20246, almost certainly by a developer who was only thinking about the affected template filters being used in HTML.

In HTML, this change is unobjectionable. In plain text, it does any number of problematic things. Of course there is no option to change this or to control this behavior. As the issue itself cheerfully notes, if you don't like this change or it causes problems, you get to write your own filter to reverse it. Nor is this documented (and the actual examples of timesince output in the documentation use real spaces).

Perhaps you might say that documenting this is unimportant. Wrong. In order to find out why this was happening to my email, I had to read the Django source. Why did I have to do that? Because in a complex system there are any number of places where this might have been happening and any number of potential causes. Django has both localization and automatic safe string quotation for things you insert in templates, so maybe this could have been one or both in action, not a deliberate but undocumented feature in timesince. In the absence of actual documentation to read, the code is the documentation and you get to read it.

(I admit that I started with the timesince filter code, since it did seem like the best bet.)

Is the new template filter I've now written sufficient to fix this? Right now, yes, but of course not necessarily in general in the future. Since all of this is undocumented, Django is not committed to anything here. It could decide to change how it generates non-breaking spaces, switch to some other Unicode character for this purpose, or whatever. Since this is changing undocumented behavior Django wouldn't even have to say anything in the release notes.

(Perhaps I should file a Django bug over at least the lack of documentation, but it strikes me as the kind of bug report that is more likely to produce arguments than fixes. And I would have to go register for the Django issue reporting system. Also, clearly this is not a particularly important issue for anyone else, since no one has reported it despite it being a three year old change.)