Wandering Thoughts archives

2015-08-15

Spam scoring systems are often not deliberately designed

In theory, my concerns about how other people's systems will react to us DKIM-signing only some of our email have a simple answer; if we don't add DMARC information that says to react to unsigned email in some way, they should do nothing. This is the spec compliant behavior and you'd have to be really obnoxious to decide to do otherwise. But that assumes that spam scoring systems are in fact deliberately designed, and my current belief is that the custom systems major email providers use are not in that sense. By that I mean that no human being sat down to write out and set up more than a small fraction of the scoring rules they use.

In today's world, one obvious path to a sophisticated spam scoring system is through various forms of statistical reasoning and machine learning (of which Bayesian spam filtering is a simple starting point). All of these techniques uncover correlations between message features and outside spam scores (as determined in various ways, such as through users telling you), and they're all blind to what those features mean as such and whether or not they 'should' be used for some purpose or interpreted in some way.

I assume that every major email provider is running such a system as part of their overall spam filtering (and there's some evidence for this in the behavior of their systems). I further assume that they're all shoveling every message feature they can get their hands on into these systems, because why not; the more features the better. I also think it's extremely likely that one of these features is DKIM information. At this point it's not particularly hard to come up with scenarios where you can objectively find correlations between things like the lack of a DKIM signature in email From: a particular domain and the likelihood of such a message being seen as spam. That there are legitimate email messages like this doesn't matter to a machine learning system any more than the fact that you're not supposed to use lack of DKIM signatures this way; all it cares about is useful correlations.

No one set out to create a system that (ab)used lack of DKIM signatures this way and the generated scoring system is not deliberately designed by anyone; the most that people did was design the machine learning meta-system that trained itself on the massive collection of accumulated message data in order to create the generated scoring system. No one understands the generated rules (even Bayesian systems are hard to peer into, never mind more sophisticated approaches) and so no one can even consider auditing them for things that shouldn't be done.

The only way to avoid having some message feature inadvertently become part of a signal deep inside a machine learning system is to exclude it. I can't make GMail's and Hotmail's and Yahoo's spam filtering systems exclude DKIM signature information from the set of message features that they train their systems on. The best I can do is not provide them with the signal in the first place by never doing DKIM signatures, making all of our email identical in this.

(Of course, by doing so I'm also sending a signal, namely the total lack of any DKIM signatures for our domains. At the moment this seems like a less dangerous signal to send for various reasons.)

(I said a much shorter version of this in a comment on my previous entry, but I feel like writing it out in full as an entry.)

spam/SpamScoringNotDeliberate written at 01:56:39; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.