Wandering Thoughts archives


A decorator for decorators that accept additional arguments

Once you're happy with functions returning other functions (technically closures), basic ordinary decorators in Python are easy enough to use, write, and understand. You get things like:

from functools import wraps
def trace(func):
    def _d(*args, **kwargs):
        print "start", func.__name__
            return func(*args, **kwargs)
            print "end", func.__name__
    return _d

def jim(a, b):
    print "in jim"
    return a + b

(If you're decorating functions that all have the same argument signature, you can make the _d() closure take the specific arguments and pass them on instead of using the *args and **kwargs approach.)

But sometimes you have decorators that want to take additional arguments (over and above the function that they get handed to decorate). The syntax for doing this when you're declaring functions that will get decorated is easy and obvious:

def fred(a, b, c):
    print "in fred:", a
    return b - c

Unfortunately the niceness stops there; an implementation of tracer() is much more complicated than trace(). Because of how Python has defined things, tracer() is no longer a decorator but a decorator factory, something that when called creates and returns the decorator that will actually be applied to fred(). You wind up with functions returning functions that return functions, or with tracer() actually being a class (so that calling it creates an instance that when called will actually do the decorating).

What we would like is for tracer() to be a regular decorator that just has extra arguments. Well, we can do that; all we need is another decorator that we will use on tracer() itself. Like so:

from functools import partial
def decor(decorator):
    def _dd(*args, **kwargs):
        return partial(decorator, *args, **kwargs)
    return _dd

def tracer(note, func):
    fname = func.__name__
    def _d(*args, **kwargs):
        print "-> %s: start %s" % (note, fname)
            return func(*args, **kwargs)
            print "-> %s: end %s" % (note, fname)
    return _d

(You can make tracer()'s arguments have the function to be decorated first, but then you have to do more work because you can't use functools.partial(). While I think that func belongs as the first argument, I don't quite feel strongly enough to give up partial().)

The one nit with this is that positional arguments really are positional and keyword arguments really are keywords. You can't, for example, write:

def fred(....):

(The only way around this is changing the order of arguments to tracer() so that func is first, which means giving up the convenience of just using partial().)

I've been thinking about decorators lately (they're probably the right solution for a code structure problem) and had this issue come up in my tentative design, so I felt like writing down my solution for later use. I'm sure that regular users of decorators already know all of these tricks.

(Decorators are one of the Python things I should use more. I don't for complex reasons that involve my history with significant Python coding.)

python/DecoratorDecorator written at 23:25:57; Add Comment

Looking at whether Zen-listed IPs keep trying to send us email

Here's a question: when an IP address listed in the Spamhaus Zen gets rejected, does it come back later or are most visits a one-time thing? This time I pulled 90 days worth of logs, extracted each day's rejections from Zen-listed IPs, and checked to see how many IPs showed up in more than one day's logs.

(Because an IP could be trying to deliver stuff right when the logs roll, the safe question is how many IPs show up in more than two days worth of logs.)

The first answer is that we have some persistent IPs but not anything that is really hammering on us. Well, at least if you look at the data this way. Here, have a table: 24 days SBL107558 20 days SBL168886 and the PBL 18 days SBL153384 15 days web.de; SBL175032 13 days web.de and SBL175032 again 10 days web.de but now SBL175030, which is basically the same as SBL175032; web.de is clearly good at getting SBL-listed. 10 days SBL157999 10 days In the PBL 9 days web.de yet again, SBL175030 9 days SBL146938
9 days No longer listed. 9 days SBL CSS 9 days SBL172432

(This table probably doesn't look that nice in the syndication feed.)

Now things get interesting, because I noticed a pattern and went digging. All of the IPs from through got rejected by us at various times in the 90 days, and all of them were rejected on multiple days. Even more interesting, the rejections stretch from day 11 through day 90 (although not continuously).

(The gaps in rejections could be either because they stopped sending to email addresses that were rejecting them, because they dropped out of Zen temporarily, or both of the above.)

This prompted me to look at /24-based reoccurrence, and there things get more interesting: 46 days One IP still in the SBL CSS 45 days 13 of 23 IPs still in the SBL CSS 43 days Nothing still listed out of the 12 IPs we rejected from this 30 days web.de, mentioned above; all four IPs still in their SBL listings 27 days SBL136747, a /24 listing dating from August 14, 2012 26 days SBL107558; one of the single IPs made it into the single-IP list 25 days Multiple IPs still in the SBL CSS 22 days Multiple IPs still in the SBL CSS

I'm going to stop here because the next '/24' is actually due to a single IP ( so we're reaching the crossover point (besides, I'm doing this all more or less by hand).

What really surprises me from looking at the by-/24 breakdown is how active the SBL CSS clearly is. If someone told me that the SBL CSS was now the largest single contributor for spam rejections, I wouldn't be surprised.

(I can't verify that without changing our mail configuration to add more logging (since SBL CSS listings expire, we'd have to capture the Zen results at the time of the actual rejection). Sadly my curiosity is not worth that.)

(This is kind of a followup to looking to see if IP addresses persist in Zen.)

Sidebar: a way in which these results may not be representative

We do Zen-based rejections only for some email addresses (only those that have opted in to it). So a Zen-listed sending IP wouldn't necessarily see continuous rejections if they kept sending to us. It depends on what email addresses they are sending to that day and they could have a day with no rejections.

I haven't tried to dig into the raw logs to see if this is happening for some of these IPs, or in general if these IPs saw a mix of successful deliveries and rejections or if they saw uniform rejections. I don't know if I'll ever do this level of analysis, since it's going past what I can easily bash together with shell scripts and awk. Past the land of shell scripts lies the land of real work.

spam/ZenRepeatHits-2013-02 written at 00:56:59; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.