GRE is a translucent tunnel

I normally expect IP tunnels to be opaque, that is to act as if they were physical links: the packet is sent down one end and pops out the other end unchanged, just as if it went over a connection between two routers, and the tunnel itself is indifferent to the details of the packets it is transporting. (The technical ISO way of describing this is that I expect IP tunnels to act entirely as layer 2 entities.)

However, one of the peculiarities of GRE is that it is a translucent tunnel, where some of the bits of the packet being tunneled show through (and are affected). In particular, GRE uses the real packet TTL.

More precisely, GRE encapsulated packets and the underlying real packets reuse each other's TTL. By default, the initial TTL of the encapsulated packet is the same as the real packet had when it got to the start point, and at the end of the tunnel the TTL of the de-encapsulated packet is whatever the TTL the encapsulated packet arrived with.

How I noticed this was trying to do a traceroute of a GRE tunneled link. Because traceroute uses the packet TTL and GRE temporarily rewrites the origin IP address, everything after my GRE gateway went blank (the TTL was expiring and the message about it was going to my GRE gateway instead of to the system running traceroute). Given that there are about 20 hops between the endpoints of the GRE tunnel, I wouldn't be surprised if it was also affecting the general reachability of the far end of the tunnel.

IPSec in transport mode uses (and alters) the regular packet headers, so GRE over transport mode IPSec is also affected by this. Tunnel mode IPSec is an opaque tunnel, so GRE over tunnel mode IPSec does not have this issue. As a result, I now have a very small IPSec tunnel.

(That tunnels temporarily rewrite the source IP address has interesting consequences for path MTU discovery; if the packet is larger than the MTU of the path the tunnel currently takes, the ICMP error packet will go to the source endpoint instead of the real source. I don't know if kernels are generally smart enough to rewrite the ICMP message a bit and send it on to the real source, if they update the tunnel MTU, or if the ICMP packet just gets dropped.)