The deprecation of FTP in browsers and its likely effects on search engines

December 6, 2020

One of the things going on in web browsers over time is that they're in the process of removing support for FTP, for instance Firefox once planned to do it this summer and Chrome may already have removed it. The obvious reason cited by Mozilla and Google for this is that use of ftp: URLs is very uncommon in web browsers (and on the web), and the FTP client implementation is a bunch of old code that must be carried around just for this. Another reason is probably that the web as a whole is increasingly moving to encrypted communications, and even if FTP theoretically supports a TLS enabled version called FTPS, in practice only a vanishingly small number of FTP sites actually support it.

As a sysadmin and someone who periodically goes digging for old documentation, I have some feelings and worries about this. The direct issue is that browsers are often one of the friendliest interfaces for digging through FTP sites; they offer convenient forward and backward navigation, visual display, and even multiple tabs (or windows). Terminal FTP clients (the general state of the art on Unix) are nowhere near as nice. However, this is the smaller of my concerns.

My larger concern is the issue of finding FTP sites, or finding that a FTP site has documentation I want. Generally I don't go to a FTP site and start hunting through it; instead, I do an Internet search and discover that some ancient thing on an old FTP site is the only source of what I want. Succeeding in these searches relies on the Internet search engines crawling and indexing FTP sites.

The major use of Internet search engines comes from browsers, and search engines are highly motivated to display only results that the browsers can actually use. If a browser can't use FTP URLs, a search engine has a reason to at least lower the priority of those URLs and may want to remove them entirely. As FTP URLs become lower and lower priority and get displayed less and less in results, search engines have less and less reasons to crawl them at all. And at the end of this process, I can no longer find old documentation on old FTP sites through web searches.

(As FTP sites stop being indexed, accessed, or usable in browsers, people also start running out of reasons to keep them operating. Many of the most valuable FTP sites for me are ones that are historical relics, and apparently survive primarily on benign neglect. Their contents are highly unlikely to be moved to HTTP sites; instead it's more likely that the contents will be discarded entirely.)

I don't expect this to happen imminently. It will probably take years before all of the infrastructure is turned off by some of the players, based on past experience. But I wouldn't be surprised if it's hard to do searches that return FTP URLs within five years, if not sooner.

Comments on this page:

By rwoodsmall at 2020-12-07 00:34:40:

Chrome has indeed disabled FTP support but it can still be enabled with a flag, at least on Chrome OS. This certainly won't last forever.

I don't believe any of these developments are good for anyone, and you're correct about the negative repercussions to findability as search and indexing are rooted out for insecure protocols, including non-S HTTP. I don't have an answer for this one, but ended up throwing together a little mod_proxy_ftp config like the below for Apache to at least be able to actually browse FTP links:

ProxyRequests On
ProxyPreserveHost On
RewriteEngine On
RewriteRule "^/ftp/(.*)$" "ftp://$1" [P]

Frankly I'm getting very, very tired of Google choosing what is best for everyone, and Mozilla quickly following suit with their us-too policy. The push toward a more secure internet is all well and good, but I'm a grown up. I know the risks with insecure FTP and HTTP and I'm willing to accept them. That's my choice, and the constant technological second guessing is getting old.

Written on 06 December 2020.
« Linux's hostname -s switch is now safe for many people, but the situation is messy
Exploring when the network is up on a machine »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Dec 6 23:49:04 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.