2024-08-14
Traceroute, firewalls, and the modern Internet: a horrible realization
The venerable traceroute command sort of reports the hops your packets take to reach a host, and in the process can reveal where your packets are getting dropped or diverted. The traditional default way that traceroute works is by sending UDP packets to a series of high UDP ports with increasing IP TTLs, and seeing where each reply comes from. If the TTL runs out on the way, traceroute gets one reply; if the packet reaches the host, traceroute gets another one (assuming that nothing is listening on the particular UDP port on the host, which usually it isn't). Most versions of traceroute can also use ICMP based probes, while some of them can also use TCP based ones.
While writing my entry on using traceroute with a fixed target port, I had a horrible realization: traceroute's UDP probes mostly won't make it through firewalls. Traceroute's UDP probes are made to a series of high UDP ports (often starting at port 33434 and counting up). Most firewalls are set to block unsolicited incoming UDP traffic by default; you normally specifically configure them to pass only some UDP traffic through to limited ports (such as port 53 for DNS queries to your DNS servers). When traceroute's UDP packets, sent to effectively random high ports, arrive at such a firewall, the firewall will discard or reject them and your traceroute will go no further.
(If you're extremely confident no one will ever run something that listens on the UDP port range, you can make your firewall friendly to traceroute by allowing through UDP ports 33434 to 33498 or so. But I wouldn't want to take that risk.)
The best way around this is probably to use ICMP for traceroute (using a fixed UDP port is more variable and not always possible). Most Unix traceroute implementations support '-I' to do this.
This matters in two situations. First, if you're asking outside people to run traceroutes to your machines and send you the results, and you have a firewall; without having them use ICMP, their traceroutes will all look like they fail to reach your machines (although you may be able to tell whether or not their packets reach your firewall). Second, if you're running traceroute against some outside machine that is (probably) behind a firewall, especially if the firewall isn't directly in front of it. In that case, your traceroute will always stop at or just before the firewall.
A note to myself about using traceroute to check for port reachability
Once upon a time, the Internet was a simple place; if you could ping some remote IP, you could probably reach it with anything. The Internet is no longer such a simple place, or rather I should say that various people's networks no longer are. These days there are a profusion of firewalls, IDS/IDR/IPS systems, and so on out there in the world, and some of them may decide to block access only to specific ports (and only some of the time). In this much more complicated world, you can want to check not just whether a machine is responding to pings, but if a machine responds to a specific port and if it doesn't, where your traffic stops.
The general question of 'where does your traffic stop' is mostly answered by the venerable traceroute. If you think there's some sort of general block, you traceroute to the target and then blame whatever is just beyond the last reported hop (assuming that you can traceroute to another IP at the same destination to determine this). I knew that traceroute normally works by sending UDP packets to 'random' ports (with manipulated (IP) TTLs, and the ports are not actually picked randomly) and then looking at what comes back, and I superstitiously remembered that you could fix the target port with the '-p' argument. This is, it turns out, not actually correct (and these days that matters).
There are several common versions of (Unix) traceroute out there; Linux, FreeBSD, and OpenBSD all use somewhat different versions. In all of them, what '-p port' actually does by itself is set the starting port, which is then incremented by one for each additional hop. So if you do 'traceroute -p 53 target', only the first hop will be probed with a UDP packet to port 53.
In Linux traceroute, you get a fixed UDP port by using the additional argument '-U'; -U by itself defaults to using port 53. Linux traceroute can also do TCP traceroutes with -T, and when you do TCP traceroutes the port is always fixed.
In OpenBSD traceroute, as far as I can see you just can't get a fixed UDP port. OpenBSD traceroute also doesn't do TCP traceroutes. On today's Internet, this is actually a potentially significant limitation, so I suspect that you most often want to try ICMP probes ('traceroute -I').
In FreeBSD traceroute, you get a fixed UDP port by turning on 'firewall evasion mode' with the '-e' argument. FreeBSD traceroute sort of supports a TCP traceroute with '-P tcp', but as the manual page says you need to see the BUGS section; it's going to be most useful if you believe your packets are getting filtered well before their destination. Using the TCP mode doesn't automatically turn on fixed port numbers, so in practice you probably want to use, for example, 'traceroute -P tcp -e -p 22 <host>' (with the port number depending on what you care about).
Having written all of this down, hopefully I will remember it for the next time it comes up (or I can look it up here, to save me reading through manual pages).