Internet routing can now vary based on things you wouldn't expect
Today Toronto had a little issue with Cloudflare, which
gave me a chance to learn a useful lesson about the modern Internet
and how it routes traffic. The summary of the lesson is that the
venerable Unix traceroute command may not be your friend any more.
I come from an era of a relatively simple Internet. Back then, the
path that your packets took through the network was expected to
depend only on the destination and the source IPs. Things in the
middle might drop some traffic or filter parts of it out, but the
path was the same whether you were using ICMP, UDP, or TCP, and
regardless of what port TCP or UDP port you were connecting to. In
this environment, ping and traceroute were reliable diagnostics
in general; if routes weren't flapping, traceroute would tell you
the path that all of your traffic was using, while ping could
tell you that the target host was there.
(If something pinged but didn't respond to the port you wanted, it was a firewall issue.)
The Cloudflare issue today did not behave like that. In particular,
plain traceroute reported one path, a short five-hop one, while
'traceroute -T -p 443' reported a rather different ten-hop path
that seemed to take a detour off to Chicago before coming back to
Toronto (and not reaching the target Cloudflare IP). At one level,
port-based routing makes a certain amount of sense; it's a lower
level version of application load balancers, and why go to all the
bother of doing complicated things just to reject UDP packets that
you don't handle. At another level it makes troubleshooting and
testing more complicated, especially for outside people. ICMP,
random UDP traffic, and actual TCP traffic to specific ports (or
emulations of it) may go to completely different places, so information
gathered in one way for one of them doesn't necessarily apply to
anything else.
Fortunately not everything is like this. Unfortunately the people who are most likely to be like this are the large cloud providers and CDNs, and those collectively host a lot of websites and places of interest (and their complexity provides more room for subtle problems).
For myself, my lesson learned from this is that if I'm trying to check
out the network path to some outside place, I should use 'traceroute -T
-p 443' (or the applicable port, but HTTPS is the most likely). Once
HTTP/3 becomes common, I'll
potentially also want to check with UDP port 443 (although that gets
complicated fast). Plain ping and traceroute are not as trustworthy
as they used to be any more.
Comments on this page:
|
|