Chris's Wiki :: blog/sysadmin/NetworkLoopWhyVanishingARP Commentshttps://utcc.utoronto.ca/~cks/space/blog/sysadmin/NetworkLoopWhyVanishingARP?atomcommentsDWiki2016-01-18T22:13:09ZRecent comments in Chris's Wiki :: blog/sysadmin/NetworkLoopWhyVanishingARP.By Aristotle Pagaltzis on /blog/sysadmin/NetworkLoopWhyVanishingARPtag:CSpace:blog/sysadmin/NetworkLoopWhyVanishingARP:4b3bc628e57325e4fad16c3af64edff60afcb102Aristotle Pagaltzishttp://plasmasturm.org/<div class="wikitext"><p>I wonder if a broader takeaway here might be that (strict?) port isolation converts some packet storm situations into vanishing packet situations.</p>
<p>(Sort of akin to the observation that garbage collection converts many crashes into memory leaks. Or more abstractly, that some architectural choices can convert a certain class of bug to another class with different severity, such that it changes the character of the system, in both operation and debugging.)</p>
</div>2016-01-18T22:13:09ZBy Chris Siebenmann on /blog/sysadmin/NetworkLoopWhyVanishingARPtag:CSpace:blog/sysadmin/NetworkLoopWhyVanishingARP:9620141450b9d0a0ea45fb057d2fc170a78efae9Chris Siebenmann<div class="wikitext"><p>Aristotle is right here, as I discovered after re-checking the behavior
we actually see on our network. For some reason I though that only
unicast packets were port isolated and broadcast packets from inside
ports went everywhere, but this is incorrect; broadcast packets from
'inside' hosts are also not propagated to other inside hosts, only up
to the outside ones. So even outside hosts will see only one extra
copy of a broadcast packet from an outside host (and inside ones will
see none).</p>
<p>(We've turned over network hardware over the past few years, so it's
possible that this behavior has changed since I first looked at it
years ago. It's certainly made our port isolated network much quieter
for end hosts than I remember it being in the past.)</p>
</div>2016-01-18T16:03:41ZBy Aristotle Pagaltzis on /blog/sysadmin/NetworkLoopWhyVanishingARPtag:CSpace:blog/sysadmin/NetworkLoopWhyVanishingARP:d9827b1d3594c87f5cf46a0d7144c83b7ebe18deAristotle Pagaltzishttp://plasmasturm.org/<div class="wikitext"><blockquote><p>what I don’t have an explanation for is why the network didn't blow up with endlessly repeated broadcast packets</p>
</blockquote>
<p>I think you just gave the reason why? I.e.: because of the port isolation. That’s if I’ve properly followed your explanation.</p>
<p>As I understood your hypothesis, it implies that <em>every</em> broadcast packet that makes it through the network will automatically get the sending port marked as “inside” – since every such packet will eventually reach the loop and get reinjected. And once it’s marked as coming from inside, it will get dropped, just like any inside-to-inside broadcast, killing off any further repetition.</p>
<p>So you should expect broadcast traffic to be amplified no more than at most twice.</p>
<p>No?</p>
</div>2016-01-17T18:39:23ZBy Ewen McNeill on /blog/sysadmin/NetworkLoopWhyVanishingARPtag:CSpace:blog/sysadmin/NetworkLoopWhyVanishingARP:6c92b633e506098f156b9fb4470888cfa26dcf4eEwen McNeill<div class="wikitext"><p>As a possible theory, broadcast packets may be software switched in the port isolated case (at least those that, eg, already need to be examined for MAC learning). This would potentially have the effect that there is a rate limit on the forwarding of things like ARP. Which in turn would stop a broadcast storm from saturating the network. (A number of modern switches also do rate limit broadcast packets.). I know that I found a client network with circulating broadcasts that were looping (despite spanning tree: a different, unknown, type of spanning tree for the devices in use), and they were noticeable but not saturating the network. Plus a noticeable switch CPU usage drop when they were drained from the network.</p>
<p>FWIW on some switches it's possible to get, eg, the MAC table via SNMP. Possibly that could be monitored for some stable MACs -- eg, the gateways.</p>
<p>Ewen</p>
</div>2016-01-17T08:33:55Z