Wandering Thoughts: Recent Entries For 2007/03/02

A story of network weirdness

We have a number of internal networks here. One of them is a port-isolated subnet for general user machines (such as Windows laptops), where the port isolation makes sure that user machines can't talk to each other and thus can't infect each other. One day, an alert user on the port isolated network reported to us that his machine was seeing packets from the outside world destined for a completely different machine.

(One of the cool things about working in a Computer Science department is that we have users that will actually notice and report this sort of thing.)

It turned out that the cause of this failure in port isolation was asymmetrical routing. The target machine had a second interface on another internal subnet, and what happened was:

1. the target machine brought up its interface on the port isolated subnet and made an active, long-lived TCP connection. This made its port-isolated IP address the origin of the connection.
2. it brought up its interface on the other subnet, and somehow made the gateway for the other subnet its default route.

   Since changing routes doesn't change the origin IP address on established connections, this created an asymmetrical route: outgoing packets for the long-lived TCP connection went out the other subnet, but incoming packets were (properly) routed back through the port isolated subnet.

3. since the switches in the port isolated subnet weren't seeing any outgoing traffic from the target machine, they started forgetting its Ethernet address to port association.

   (But because it was an active connection, the IP address to Ethernet address mapping stayed in the router's ARP cache.)

When a switch doesn't know what port is associated with the destination Ethernet address of an incoming packet, it broadcasts the packet to all ports. In short order, packets for the target machine were being flooded to every port in our entire port isolated subnet, where one alert user noticed the strange traffic.

This wouldn't have happened with a less active connection, because the router's ARP cache would have timed out, forcing an ARP broadcast, causing the target machine to reply over its interface on the port isolated subnet, causing the switches to (re)learn the necessary Ethernet address to port associations.