Wandering Thoughts archives

The different between an SMTP proxy and a SMTP relay

This is one of those distinctions that is both obvious and obscure.

SMTP proxies and SMTP relays both sit between two machines; call them the outside client and the inside mail server, as in the case of our external mail gateway.

A SMTP proxy is synchronous. When it gets a SMTP connection from the outside client, it establishes an SMTP conversation with the inside mail server. When it receives a SMTP command from the outside client, it applies its own processing (if any), sends the command on to the inside mail server, and only gives a positive reply to the outside client if the inside mail server also accepts it. Of course a filtering SMTP proxy can reject a SMTP command without even bothering the inside mail server if it wants, and it can filter and modify the commands it sends onward (and the replies it gets back).

('Command' here is broadly construed; for example, all of the message body might be seen as a single command.)

A SMTP relay is asynchronous. Even if it immediately starts a SMTP conversation with the inside mail server, it won't delay replies to the outside client's SMTP commands until it has answers from the inside mail server. Most SMTP relays don't even start talking to the inside mail server until they've had a full conversation with the outside client.

SMTP proxies are in many ways simpler than SMTP relays, because they never becomes responsible for messages themselves; responsibility passes directly from the outside client to the inside mail server, because the SMTP proxy never tells the outside client that the mail has been accepted until the inside mail server has also accepted it. A SMTP proxy does not have to reliably save messages to disk in case they can't be immediately delivered or the system crashes, it does not have to maintain and retry queues of pending messages, and it has the important property (from an antispam perspective) that it never has to generate and send a bounce message and thus it never has to worry about the accept-then-bounce issue.

Because a SMTP relay actively takes responsibility for messages, it does have to deal with all of those issues. (It may chose to break the rules, in which case mail can silently vanish.)

Essentially all MTAs are SMTP relays, because that's their purpose; they accept and transport and store email for people. Very few are capable of acting as SMTP proxies, because this is a rare case for them.

Conversely, anti-spam filters that you talk to using SMTP are generally SMTP proxies, because they don't want to deal with all of the extra complexity of being an SMTP relay (especially the the accept-then-bounce issue).

The two sorts of Referer spam that I see

In general, there are two sorts of Referer spam that I see here on WanderingThoughts. The first sort is what I'll call 'plain referer spam', HTTP requests that have Referer URLs that are various spam websites. Many of these are easily recognizable because they are of the form 'http://web.site', with no trailing slash on the end; this is a legal URL, but it is not one that regular browsers ever send. Usually the domain names or the rest of the URL make it pretty clear that this is spam instead of a crazy browser.

(You can enter a website name without the trailing slash, but your browser will almost always add the trailing slash when it actually visits the page. Then when you click on links on the page, it is the slash-added URL that is sent as the Referer.)

The other sort is what I call 'active text', and is much less common. Active text is snippets of actual text in the Referer instead of an URL; I believe I've seen HTML, BBCode, and even an attempt with JavaScript. I find this spam both depressing and intensely irritating, especially the JavaScript. It's depressing because that spammers are doing it implies that there is software that just puts the text from Referer headers on some web page without escaping it first (which is terribly wrong); it's irritating because it's clearly attempting to exploit a security flaw and I'm a sysadmin.

(These Referer headers are clearly bogus; I don't think they even bother pretending by, say, putting a 'http://...' on the front of a garbage word at the start.)

My memory, along with some early entries here, suggests that Referer spam used to be a lot more common than it is now. These days a typical volume is at most one or two attempts a day, and I'm pretty sure that there are days with none at all. Since the spammers are not hitting me with the same Referer over and over, I generally assume that their current goal is to collect links from as many web pages as possible in order to boost their search engine rankings. Naturally they are completely indifferent to the fact that CSpace has no page that shows Referer information, which makes their efforts here completely useless.

(Like email spam, web spam has become something that I no longer pay much attention to unless it is really glaring and in my face.)

PS: if you have public Referer logs, I sugges that you turn them off. If you have private Referer logs that are presented in any sort of web interface, I suggest that you make sure that your software properly quotes and escapes the Referer information. I view my Referer logs in plain text with minimal processing, but I'm aware that this is attractive only to crazy people like me.

(And never, ever investigate a spammed website with anything except an extremely secure browser configuration; you should assume that all of them are absolutely crawling with infectious malware. I use lynx, or sometimes wless.)