A browser tweak for system administrators doing (web) network debugging

January 12, 2023

As a system administrator (and sometimes an ordinary user of the web), I periodically find myself trying to work out why I or people around here can't connect to some website or, sometimes, a portion of the website doesn't work. It turns out that there's a tweak you can make to Firefox and Chrome (and probably other browsers) that makes this somewhat easier to troubleshoot.

(We once had an incident where Google Cloud Platform stopped talking to some of our IPs. Some websites host only a portion of their assets or web application in GCP, so people could load a website's front pages (hosted outside of GCP) but trying to go further or do things in the web app would fail (when it touched GCP and GCP said 'nope'). Even figuring out what was going on took some people here rather a lot of work.)

Modern web browsers have a 'Web Developer Tools' environment that includes a Network tab that will tell you about the requests the current page is doing. By default the information the Network tab presents is focused on the interests of web developers and so lacks some information that system administrators find very helpful. However, you can customize it, and in particular you can make it also show the (HTTP) protocol being used and the remote IP, which are extremely useful for people like me.

To do this, call up Web Developer Tools with, for example, Ctrl+Shift+I. Switch to the Network tab if you're not already on it, and make a request so that the tab displays some data and you have the header. Right click on the header and turn on the Protocol and the remote IP. Turning on the Scheme is optional (it will probably mostly be 'https') but will let you spot websocket connections if you want to check or verify that you have one. Knowing the HTTP protocol is important these days because HTTP/3 is an entirely different transport and may run into firewall issues that HTTP/2 and HTTP/1.1 don't.

(This isn't relevant if you've turned HTTP/3 off in your browser, but then your users probably don't have it turned off and you may need to emulate their setup.)

In an ideal world there would be a way to get your browser to tell you about all currently open or in-flight network connections, both the low level details of where you're connecting to (and how) and the high level details of what protocol the browser is trying to speak over the connection, what web request it's trying to satisfy, and so on. Firefox has about:networking, but generally this gives only low-level details in a useful form. Chrome has taking and exporting logs in chrome://net-export and there's also chrome://net-internals (but it didn't do much for me), and maybe there's other things lurking in chrome://chrome-urls.

(In Firefox, for example, I can see that Firefox is holding open a HTTPS connection to an AWS IP and periodically doing stuff with it, and tcpdump confirms this, but about:networking won't tell me what host name this is for or what web page it's associated with. This is probably some Mozilla internal service, but finding out that it might be 'push.services.mozilla.com' took an absurd amount of work.)

(All of this was sparked by an issue I incorrectly blamed on HTTP/3, which led me to the Cloudflare blog on how to test HTTP/3, which taught me about the Web Developer Tools trick for the protocol.)

PS: Firefox will at least let you get a global view of (new) network activity that it knows about, via the Network tab of the "Browser Toolbox" (Ctrl+Shift+Alt+I). You want to pick 'Multiprocess (slower)'. I believe this will also let you temporarily disable the cache globally, across all windows and tabs.

Written on 12 January 2023.
« Sometimes it actually is a kernel bug: bind() in Linux 6.0.16
Ubuntu 22.04 LTS servers and phased apt updates »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Jan 12 23:17:44 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.