Wandering Thoughts archives


We need to deploy anti-spam precautions even if they're a bit imperfect

A few years ago we had a local spam incident. In its wake, we made some configuration changes and started exploring things like ratelimiting outgoing email. Our first step in this was to set our Exim configuration to track rate limits without enforcing them, so that we could figure out what limits to set that would stop spammers without causing problems for our users.

At one level, this was a sensible decision. Causing disruptions to our users might create political pressure that would stop us from taking any precautions against future spam runs from compromised local accounts. But, well, we never really found a level that our users didn't run over once in a blue moon, and when the overruns only happen once in a blue moon it's hard to iterate on tuning limits. And so things sat there from 2012 until today.

Today we had another little local spam incident (as people on Twitter might have guessed). Our ratelimit tracking code dutifully logged that this was happening and that our hypothetical ratelimits were being exceeded, and by quite a bit too:

[...] Warning: SENDER RATE LIMIT HIT: 27943.5 / 60m max 200 / [...]

So. Yeah.

It may not surprise you to hear that now we have some active ratelimits; in fact we more or less simply made our previous tracking ratelimits into enforcing ones. They're undoubtedly not perfect ratelimits, and in fact I'm fairly sure that within six months someone here sending out an entirely legitimate burst of email will run into them. But as usual the perfect is the enemy of the good. Our quest to deploy only perfect anti-spam precautions that would never inconvenience our users turned out to result in us deploying almost no anti-spam precautions, with regrettable results.

(Nor did we avoid inconveniencing users, since some of them had email bounce due to the machine in question temporarily picking up a bad sender reputation.)

We don't want to deploy significantly imperfect anti-spam precautions, for obvious reasons. Something that gets in the way of our users on a frequent basis is no good. But I've come around to the view that we need to be more willing to deploy things that are a bit imperfect and then sort out the problems when they happen. Otherwise, well, we may be looking at something like this happening all over again.

(One of those may be some sort of scanning of our outgoing email, or at least some of it. Despite my historical reservations, I now think it's possible to do this in a good way and I think that the risks of false positives may be one of those 'a bit imperfect' things we can live with, at least initially. But right now I'm kind of thinking out loud in the immediate aftermath of an incident, which gives me some biases.)

spam/DeployImperfectAntispamPrecautions written at 23:54:33; Add Comment

Adding a new template filter in Django 1.9, and a template tag annoyance

As the result of my discovery about Django's timesince introducing nonbreaking spaces, I wanted to fix this. Fixing this requires coding up a new template filter and then wiring it into Django, which took me a little bit of flailing around. I specifically picked Django 1.9 as my target, because 1.9 supports making your new template filters and tags available by default without a '{% load ... %}' statement and this matters to us.

When you are load'ing new template widgets, your files have to go in a specific and somewhat annoying place in your Django app. Since I wasn't doing this, I was free to shove my code into a normal .py file. My minimal filter is:

from django import template
from django.template.defaultfilters import stringfilter

register = template.Library()

def denonbreak(value):
   """Replace non-breaking spaces with plain spaces."""
   return value.replace(u"\xa0", u" ")

The resulting filter is called denonbreak. Although the documentation doesn't say so explicitly, you are specifically handed a Unicode string and so interacting with it using plain strings may not be reliable (or work at all). I suppose this is not surprising (and people using Python 3 expect that anyways).

To add your filter(s) and tag(s) as builtins, you make use of a new Django 1.9 feature in the normal template backend when setting things up in settings.py. This is easiest to show:

    'BACKEND': 'django.template.backends.django.DjangoTemplates',
    'OPTIONS': {
       'builtins': ['accounts.tmplfilters'],

(Do not get diverted to 'libraries'; it is for something else.)

At this point you might ask why I care about not needing to {% load %} my filter. The answer is one of the features of Django templates, which is that there is no good way to suppress newlines at the end of template directives.

Suppose you have a template where you want to use your new tag:

{% load something %}
The following pending account requests haven't been
handled for at least {{cutoff|timesince|denonbreak}}:

Django will remove the {% load %}, but it won't remove the newline after it. Thus your rendered template will wind up starting with a blank line. In HTML this is no problem; surplus blank lines silently disappear when the browser renders the page. But in plain text it's another story, because now that newline is sticking around, clearly visible and often ugly. To fix it you must stick the {% load %} at the start of the first real line of text, which looks ugly in the actual template.

({% if %} is another template tag that will bite you in plaintext because of this. Basically any structuring tag will. I really wish Django had an option to suppress the trailing newline in these cases, but as far as I know it doesn't.)

This issue is why I was willing to jump to Django 1.9 and use the 'builtins' feature, despite what everyone generally says about making custom things be builtins. I just hate what happens to plaintext templates otherwise. Ours are ugly enough as it is because of other tags with this issue.

python/Django19NewTemplateFilter written at 01:19:34; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.