Today Toronto had a little issue with Cloudflare, which gave me a chance to learn a useful lesson about the modern Internet and how it routes traffic. The summary of the lesson is that the venerable Unix traceroute command may not be your friend any more.

I come from an era of a relatively simple Internet. Back then, the path that your packets took through the network was expected to depend only on the destination and the source IPs. Things in the middle might drop some traffic or filter parts of it out, but the path was the same whether you were using ICMP, UDP, or TCP, and regardless of what port TCP or UDP port you were connecting to. In this environment, ping and traceroute were reliable diagnostics in general; if routes weren't flapping, traceroute would tell you the path that all of your traffic was using, while ping could tell you that the target host was there.

(If something pinged but didn't respond to the port you wanted, it was a firewall issue.)

The Cloudflare issue today did not behave like that. In particular, plain traceroute reported one path, a short five-hop one, while 'traceroute -T -p 443' reported a rather different ten-hop path that seemed to take a detour off to Chicago before coming back to Toronto (and not reaching the target Cloudflare IP). At one level, port-based routing makes a certain amount of sense; it's a lower level version of application load balancers, and why go to all the bother of doing complicated things just to reject UDP packets that you don't handle. At another level it makes troubleshooting and testing more complicated, especially for outside people. ICMP, random UDP traffic, and actual TCP traffic to specific ports (or emulations of it) may go to completely different places, so information gathered in one way for one of them doesn't necessarily apply to anything else.

Fortunately not everything is like this. Unfortunately the people who are most likely to be like this are the large cloud providers and CDNs, and those collectively host a lot of websites and places of interest (and their complexity provides more room for subtle problems).

For myself, my lesson learned from this is that if I'm trying to check out the network path to some outside place, I should use 'traceroute -T -p 443' (or the applicable port, but HTTPS is the most likely). Once HTTP/3 becomes common, I'll potentially also want to check with UDP port 443 (although that gets complicated fast). Plain ping and traceroute are not as trustworthy as they used to be any more.