Chris's Wiki :: blog/tech/InternetPathsMayVaryByPort Commentshttps://utcc.utoronto.ca/~cks/space/blog/tech/InternetPathsMayVaryByPort?atomcommentsDWiki2021-03-31T15:23:11ZRecent comments in Chris's Wiki :: blog/tech/InternetPathsMayVaryByPort.By Chris Siebenmann on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:a46144a1d8942f71d6584342aaebac6322610266Chris Siebenmann<div class="wikitext"><p>In an additional surprise, the mere presence of a +subnet argument to
dig appears to change what answer I get from Facebook's servers, even if
I provide the same /24 as I'm querying from. This is slightly annoying
as it means I can't map out the range that Facebook is supplying the
different answer for.</p>
<p>My large scale lesson is that how big places do their DNS and what
answers you'll get from them is both unpredictable and variable from
outside. Facebook is clearly doing something, but only they know exactly
what. The corollary of this is that if users have problems reaching some
big outside site, we'd better ask them what IP address the site resolves
to for them because it may not be the answer we get.</p>
</div>2021-03-31T15:23:11ZBy Chris Siebenmann on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:bfee2d4e42684f117bf4a3d9be3988f5a4137da3Chris Siebenmann<div class="wikitext"><p>There were separate DNS resolvers on the different subnets. I also
tested this with direct <code>dig</code> queries to Facebook's listed NS servers
and they answered this way (of course, direct queries expose the
subnet). However, EDNS Client Subnet does create an interesting test:
I can see if supplying a different network's subnet changes the answer
that Facebook gives. The answer is yes; a query from our subnet with
'<code>dig +subnet=<other>/24</code>' gets a different answer than without +subnet.
So this appears to be Facebook deciding that different subnets here get
different servers, not ECMP diverting traffic to different DNS servers
that give us different answers.</p>
</div>2021-03-31T15:16:19ZFrom 193.219.181.219 on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:1d485838a04666c6a4d00b2abdf30f1f4f615accFrom 193.219.181.219<div class="wikitext"><blockquote><p>As a pragmatic example, we had a puzzling case where Facebook's DNS servers gave different answers to different /24s within the university. It didn't occur to me until now that ECMP is one possible explanation of this.</p>
</blockquote>
<p>Hmm, if you mean your own recursors from different /24s were talking to different Facebook authoritative servers, it might be the case due to ECMP.</p>
<p>But this also reminds me of the "EDNS Client Subnet" extension that DNS resolvers now often add to forwarded queries, so that even if the entire university shares the same set of resolvers, they might still inform upstream authoritative servers about which /24 each query came from.</p>
</div>2021-03-31T10:43:27ZBy Chris Siebenmann on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:23e58c4159bb1c88c7de8c7cd4bd19edc88b1d69Chris Siebenmann<div class="wikitext"><p>I think that sysadmins have generally assumed that traceroutes from hosts
in the same /24 or perhaps more generally the same ASN would be routed
more or less the same way (I certainly have). ECMP makes this not true
even for adjacent IPs on the same /24.</p>
<p>As a pragmatic example, we had a puzzling case where Facebook's
DNS servers gave different answers to different /24s within the
university. It didn't occur to me until now that ECMP is one possible
explanation of this.</p>
<p>(Now that I check, this DNS separation is still happening.)</p>
</div>2021-03-29T14:07:40ZBy Aristotle Pagaltzis on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:c4c04fb257e385eef01eb90394f0e8ed84f9b2d2Aristotle Pagaltzishttp://plasmasturm.org/<div class="wikitext"><blockquote><p>[A] traceroute from an adjacent host […] may well be taking a different path in some situations.</p>
</blockquote>
<p>That seems a given, and an always-been-a-given at that, no? Any time you run a traceroute from somewhere else, you might get results that don’t apply to your host of interest, if only for such mundane reasons as diverging networking configurations. So even with instincts harking from a simpler time on the internet, you would expect or at least not be surprised by that.</p>
<p>The newer reality Chris points out, that even traceroute from the <em>same</em> machine is increasingly likely to produce inapplicable and misleading results, is rather more dismaying.</p>
</div>2021-03-29T10:15:01ZFrom 193.219.181.219 on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:e27fa349aef15be8df299c4e3b8dde2a7d38b449From 193.219.181.219<div class="wikitext"><p>Oh, and this obviously also means that your source IP has an effect as well. For example, odd IP addresses may hash down to one upstream and even IP addresses to another.</p>
<p>So if you're figuring out connectivity issues on one host <code>x.y.z.6</code> but run a traceroute from an adjacent host <code>x.y.z.7</code> (e.g. because that's what you happened to have a ssh session to), it may well be taking a different path in some situations.</p>
</div>2021-03-28T09:15:48ZFrom 193.219.181.219 on /blog/tech/InternetPathsMayVaryByPorttag:CSpace:blog/tech/InternetPathsMayVaryByPort:f87b1db8dda1d2b1717a02a7d5fe5566d46060c9From 193.219.181.219<div class="wikitext"><p>One thing about traceroute and mtr that already existed before Cloudflare is that some ISPs use ECMP routing (equal cost multipath), where the same destination just has more than 1 gateway of identical preference. The gateway for each packet is then chosen by hashing layer-3 and often layer-4 headers. It isn't port-based routing, exactly, but the result is influenced by them.</p>
<p>(In Linux terms, they might have something like <code>ip route add default nexthop via 192.168.1.1 nexthop via 192.168.1.2</code> etc.)</p>
<p>Which means, in the default traceroute mode where it probes successively higher UDP ports for each hop, it may actually end up choosing a different path for each probe, and may show nonsensical results with both paths mixed in. (In mtr this is especially obvious.) Specifying ICMP or at least a fixed UDP probe port then becomes necessary to avoid this.</p>
</div>2021-03-28T09:12:34Z