Wandering Thoughts archives

2021-02-25

The HTTP Referer header is fading away (at least as a useful thing)

The HTTP Referer header on requests is famously misspelled (it should be Referrer), and also famously not liked because of privacy and security concerns. The privacy and security concerns are especially strong with external ('cross-origin') Referers, which is also the ones that many people find most useful because they tell you where visitors to your pages are coming from and let you find places where people have linked to you or are mentioning you.

I've been reading my Referer logs for essentially as long as Wandering Thoughts has existed, and over the years (and especially lately) it's become clear to me that the Referer header is fading away. Fewer requests have Referer headers, and many of the values that are there aren't all that useful (at least to me). Some of this is the general issue of social media and web applications, where most everything from a place like Twitter either has 'https://twitter.com/' (if the person came from web twitter) or nothing (if they came from a Twitter client). Others seem to be specific choices made by sites. For example, a lot of search engines now arrange things so that the Referer you see is their main URL and doesn't have any information on what the person searched for that led them to your pages.

(Probably an increasing number of people are also using browser extensions that block or spoof Referer, although I don't know if this is common.)

Referer is clearly going to fade away more in the future. This effort started with the Referrer-Policy header, which gave web server operators a simple way to mostly control the outbound Referer from any links on their web pages (without having to touch the HTML). Now the browsers are in the process of moving to a stricter default behavior, called 'strict-origin-when-cross-origin'; this sends only the origin (the website), omitting the path and the query string. A switch to this default would make almost all websites behave the way that Twitter and some search engines do (although for different reasons).

In theory web sites could set a Referrer-Policy to revert back to the current state of affairs once the browser default changes. In practice most websites will never change the default (and many of the ones that do might make it stricter, likely going to 'same-origin'). And someday browsers will probably ratchet privacy and security one step further, so that by default they only send Referer headers to the same site.

When that happens, I'll definitely miss the old days when I could see where my visitors were coming from. Discovering various articles and people who've linked to my entries has definitely made me happy and sometimes given me valuable information and feedback. But the arrow of time on the web points to more privacy, and there's not much we can do about that; the bad actors vastly outnumber the good ones.

web/FadingHTTPReferer written at 23:35:33; Add Comment


Page tools: See As Normal.
Search:
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.