I'm considering ways to mass-add URLs to Firefox's history database

April 30, 2015

I wrote yesterday about how I keep my browser history forever, because it represents the memory of what I've read. A corollary of this is that it bugs me if things I've read don't show up as visited URLs. For example, if all of the blog entries and so on here at Wandering Thoughts were to turn unvisited tomorrow, that'd make me twitch every time I read something here and saw a blue link that should instead be visited purple.

(One of the reasons for this is that links showing visited purple is a sign that they point to the right place. Under normal circumstances, if links on Wandering Thoughts suddenly go blue, something has probably broken. And when I'm drafting entries, a nominal link to an older entry that shows blue is a sign that I got the link wrong.)

Which winds up with the problem: Wandering Thoughts and indeed this entire site is in the process of moving from HTTP to HTTPS. The HTTP versions of all of the entries and so on are in my Firefox history database, but Firefox properly considers the HTTPS version to be a completely different URL and so not in the history. So, all of a sudden, all of my entries and links and so on are unvisited blue. At one level this is not a problem. After all, I know that I've read them all (I wrote them). In theory, I could leave everything here alone, then maybe re-visit links one by one as I use them in new entries or otherwise run across them. But the whole situation bugs me; by now, seeing all the links be purple is reassuring and the way things should be, while blue links here make me twitch.

Conceptually the fix is simple. All I have to do is get every HTTP URL for here out of my existing history database, mechanically turn the 'http:' into 'https:', and then add all of the new URLs to Firefox's history database. All of the last visited and so on values can be exactly copied from the HTTP version of the URL. The only problem is that as far as I know there is no tool or extension for doing this.

(There are plenty of addons for removing history entries, which is of course exactly the opposite of what I want.)

These days, Firefox's history in is a SQLite database (places.sqlite in your profile directory). There are plenty of tools and packages to manipulate SQLite databases, which leaves me with merely the problem of figuring out what actually goes into a history entry in concrete detail (and then calculating everything that isn't obvious). So all of this is achievable, but on the other hand it's clearly going to be a bunch of work.

(While the Places database is documented, parts of this documentation are out of date. In particular, current Firefox places.sqlite has a unique guid field in the moz_places table.)

PS: The other obvious nefarious hack is to literally rewrite the URLs in all current history entries to be 'https:' instead of 'http:', possibly by dumping and then reloading the moz_places table. Assuming that you can change the URL schema without invalidating any linkages in the database, this is simple. Unfortunately it has a brute force inelegance that makes me grumpy; it's clearly the expedient fix instead of the right one.

Written on 30 April 2015.
« Why I have a perpetual browser history
OmniOS as a NFS server has problems with sustained write loads »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 30 23:36:51 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.