The fading HTTP Referer header and (Google) Search paywall bypasses
There are a lot of newspaper and other media places that have a general paywall (where you must be a subscriber to see their content), but also make an exception to this paywall if the visitor is coming directly from an Internet search (or at least a Google search; I don't know if these places will let visitors from other search engines in). Today, how these places generally know that you're coming from an Internet search is the HTTP Referer header that your browser puts on your request when you follow the link from the search page. That would be the same HTTP Referer header that's fading away, fundamentally because browsers don't like it.
Today it occurred to me that this creates some interesting issues
both for Internet search engines and for places currently using a
permeable paywall. For Internet search engines, it probably makes
them willing to set an explicit
when browsers start defaulting to something inconvenient for them
(Google Search and Bing don't appear to do this today). Search
engines get a variety of things out of visibly sending traffic to
media sites, or really out of visibly sending traffic to anywhere,
so they have an incentive to make this traffic visible.
(I'd say that search engines have additional options if browsers refuse
to cooperate, but the company behind the dominant Internet search engine
is also the company behind the dominant Internet browser, so there's no
risk of that. Chrome is going to do what is useful for Google, including
If Internet search engines keep sending
Referer headers, then
places with permeable paywalls may not need to do anything. In
theory, as the default
changes, such places will need to live without knowing the terms
that their visitors searched for. In practice I suspect that they
already see a lot of visits with only the origin (the website, eg
'https://google.com') and no search terms, because that's what I
do here. If for some reason the Internet search engines
Referer entirely and won't do anything else to signal
the origin, such as adding special query parameters to the URL, then who knows. Perhaps the paywalls will
get less permeable than they are today. If nothing else, such a
development would be interesting to see.
(Before I started thinking through what search engines would likely do, I guessed that publishers would wind up with a real problem. Now I think they probably will be able to just keep on as usual, because search engines are likely to take steps to keep things running as normal.)
Comments on this page:Written on 16 March 2021.