Wandering Thoughts archives

How (some) syndication feed readers deal with HTTP to HTTPS redirections

It's now been a bit over a year since Wandering Thoughts switched from HTTP to HTTPS, of course with a pragmatic permanent redirect from the HTTP version to the HTTPS version. In theory syndication feed readers should notice permanent HTTP redirections and update their feed fetching information to just get the feed from its new location (although there are downsides to doing this too rapidly).

In practice, apparently, not so much. Looking at yesterday's stats from this server, there are 6,700 HTTP requests for Atom feeds from 520 different IP addresses. Right away we can tell that a number of these IPs made a lot of requests, so they're clearly not updating their feed source information. Out of those IPs, 30 of them did not make HTTPS requests for my Atom feeds; in other words, they didn't even follow the redirection, much less update their feed source information. The good news is that these IPs are only responsible for 102 feed fetch attempts, and that a decent number of these come from Googlebot, Google Feedfetcher (yes, still), and another web spider of uncertain providence and intentions. The bad news is that this appears to include some real syndication feed readers, based on their user agents, including Planet Sysadmin (which is using 'Planet/1.0'), Superfeedr, and Feedly.

The IPs that did at least seem to follow the HTTP redirection have a pretty wide variety of user agents. The good or bad news is that this includes a number of popular syndication feed readers. It's good that they're at least following the HTTP redirection, but it's bad that they're both popular and not updating feed source information after over a year of permanent HTTP redirections. Some of these feed readers include CommaFeed, NewsBlur, NetNewsWire, rss2email, SlackBot, newsbeuter, Feedbin, Digg's Feed Fetcher, Gwene, Akregator, and Tiny Tiny RSS (which has given me some heartburn before). Really, I think it's safer to assume that basically no feed readers ever update their feed source information on HTTP redirections.

As it turns out, the list of user agents here comes with a caveat. See the sidebar.

(Since it's been more than a year, I have no way to tell how many feed readers did update their feed source information. Some of the people directly fetching the HTTPS feeds may have updated, but certainly at least some of them are new subscribers I've picked up over the past year.)

At one level, this failure to update the feed source is harmless; the HTTP to HTTPS redirection here can and will continue basically forever without any problems. At another level it worries me, both for Wandering Thoughts and for blogs in general, because very few things on the web are forever and anything that makes it harder to move blogs around is worth concern. Blogs do move, and very few are going to be able to have a trail of HTTP redirections that lives forever.

(Of course the really brave way to move a blog is to just start a new one and announce it on the old one. That way it takes active interest for people to keep reading you; you'll lose the ones who aren't actually reading more (but haven't removed you from their feed reader) and the ones who decide they're not interested enough.)

Sidebar: Some imprecision in these results

Without more work than I'm willing to put in, I can't tell when a HTTPS request from a given IP is made due to following a redirection from a HTTP request. All I can say is that an IP address that made one or more HTTP requests also made some HTTPS requests. I did some spot checks (correlating the times of some requests from specific IPs) and they did look like HTTP redirections being followed, but this is far from complete.

The most likely place where I'd be missing a feed reader that doesn't follow redirections is shared feed reader services (ie, replacements for Google Reader). There it would be easy for one person to have imported the HTTP version of my feed and another person to have added the HTTPS version later, quite likely causing the same IP fetching both HTTPS and HTTP versions of my feed and leading me to conclude that it did follow the redirection.

I have some evidence that there is some amount of this sort of feed duplication, because as far as I can tell I see more HTTPS requests from these IPs than I do HTTP ones. Assuming my shell commands based analysis is correct, I see a number of cases where per-IP request counts are different, in both directions (more HTTPS than HTTP, more HTTP than HTTPS).

(This is where it would be really useful to be able to pull all of these Apache logs into a SQL database in some structured form so I could sophisticated ad-hoc queries, instead of trying to do it with hacky, awkward shell commands that aren't really built for this.)

My concern about the potential dominance of the mobile web

Yesterday I wrote about how I don't know and can't find out how dominant the mobile web actually is. Today, let's take the (implicit) claims of the 'mobile first' design people as given and posit that the mobile web is either already dominant or going to become dominant in the not too distant future. In that case, well, I have concerns. In fact they are predictable concerns and are basically wrapped up in that nice phrase of 'mobile first' design.

Put simply, the web has seen this show before but back then it was usually called 'IE first design' or before then 'Netscape first design'. What these really translate to is that everyone else is a de facto second class citizen. Oh, sure, some amount of designers have good intentions and want to do a good job for other people (like the desktop web), but designers (and web developers) also have deadlines and priorities and only so much time. When you say 'mobile first', you implicitly accept that desktop people are allowed to have a degraded experience. That's what putting someone second means.

This is a direct concern of mine because I'm exclusively a desktop user, and more than that a Linux user of Firefox. This puts me well outside even the outside area of a broad 'mobile first' design (which is likely to target Safari and Chrome). This goes beyond worries about non-functional websites, as mobile first designs are likely going to be built for significantly smaller screen sizes than I have (well, smaller browser window sizes than I have). A functional website that wastes three quarters of the screen space I have and squeezes all of the content into a small area is probably not going to make me very happy.

(In theory responsive design can deal with this. In practice, I'm not at all convinced that people can (or will) make designs that scale across that large a screen size change, especially if mobile becomes their first priority.)

All of this concern is probably somewhat overblown, for many reasons including that much of my use of the web is fundamentally a diversion. In fact, the growth of the mobile web may have seen off the largest threat to the usable web that I've faced, namely the growth of Javascript-mandatory websites. Those defy even 'view page in no style' or Firefox's reader view.

(My understanding is that JavaScript on mobile is a lot slower and more power-hungry than it is on desktop, especially on lower end smartphones. My impression is that this has led to less client side JS and more server side content rendering in mobile focused website designs, although I may be out of touch here.)

How dominant is the mobile web (versus the desktop web)?

I was all set to write an entry about how the increasing dominance of the 'mobile web' (web usage from mobile devices) concerned me, but then I made the mistake of trying to find some reasonably current and solid facts about how web traffic is split between mobile and desktop and how it's been changing. Now I'm writing a different entry, because when I started looking I couldn't find anything that felt solid enough.

Everyone seems to agree that mobile usage as a whole is quite high (at least in first world countries), but a bunch of sources also claim that a lot of that time is spent in apps, not in browsing the web from mobile devices. Google apparently said in 2015 that more searches came from mobile devices than desktops in a number of countries (including the US and Japan). But the only actual usage numbers I could find seem to still have mobile browsing under half of browsing, and apparently not changing too fast; however, I have no idea how trustworthy those numbers are.

(Most people who put forward numbers seem to be out to sell you something, and who knows how reliable their methodologies are. I do believe Google's claim, but they did carefully hedge it.)

There is certainly a 'mobile first' movement for website design, but again there seems to be an aspect of boosterism to it. Certainly there have been plenty of past web design 'trends' that people advocated quite hard for but that never materialized in practice to the degree that their boosters thought (semantic markup, anyone?). And to the extent that this material is produced by pundits, well, pundits have to have something to hold forth about and that's often a future trend instead of a present reality.

(Some of this is a self-fulfilling prophecy, too. Depending on your perspective and how the website works, a 'mobile first' website might either unleash the pent up demand for well-done content from mobile devices or drive away some amount of the remaining desktop usage.)

Sidebar: 'Important' versus 'dominant'

If the mobile web is approaching half of all your web traffic, it's certainly big enough to be worth catering to and designing for. But if it's only (less than) half of traffic you can't focus on it to the exclusion of desktop design, the way you might if desktop usage was (say) only 20% and decreasing.

(It seems pretty clear that including the mobile web in your design is important these days. If nothing else, Google will rap your knuckles in search results if they don't consider you mobile-friendly (although some sources say that that's only for mobile searches).)