2023-10-23
Internet network sessions can be remarkably durable
These days, we do NAT for our internal networks on our perimeter firewall instead of the firewalls for these internal networks. One of the advantages of doing your NAT on your perimeter firewall is that the external NAT IP ranges you use can be on a completely different subnet than your internal firewalls, which is quite handy if you want to do things like round robin NAT but are running out of IP address space on the subnet with your firewalls. With OpenBSD PF based firewalls, you can normally switch your NAT IPs without breaking existing connections (provided that you have state entries, which is the normal behavior for good reasons). However, if you do this live switch, the old state table entries for the old NAT IPs will stick around for as long as the underlying connections or sessions last, which makes reusing those old NAT IPs a little bit exciting. Probably you want to wait until the sessions and thus the state table entries go away on their own.
We recently did such a switch for our public NAT IPs, especially for the IP ranges used for round robin NAT. When we did this, I was expecting the state tables to empty out relatively rapidly. It turned out that I was wrong, and not just due to things like SSH connections. We saw a non-trivial number of long lived state table entries for HTTPS and even HTTP connections (or at least TCP connections to port 80 and port 443), along with some UDP sessions, external SSH sessions, and the like. Some kept on going for more than a week after the change (and we don't know how long they'd been going before then).
At one level this is unsurprising. Modern web applications can hold open long-lived HTTP requests or WebSockets (also) connections in order to do things like receive (near) real time updates in messaging and chat applications. Checking 'about:networking#sockets' in my Firefox generally shows at least one active socket even if I'm not doing anything. At another level the sheer durability of these connections across the Internet surprised me, because a part of my mind is still stuck back in the era of somewhat chancy Internet connectivity, when you could expect something to happen to your packets sooner or later and HTTP's essentially stateless ability to retry was an important virtue.
In general it's great to see that the Internet can be that stable and reliable these days, even if it was slightly inconvenient for us being able to immediately reuse those old NAT IPs. Of course there's some survival bias here; I don't know how many ongoing connections didn't survive that long, only that some did.
(Some of the connections will have ended naturally because of events on our side, like the person closing their browser or ending their session on whatever website they were on.)