A limitation of tcpdump is that you can't tell in from out

January 18, 2016

Suppose that you are running tcpdump on a network that's experiencing problems, on a machine which you know is supposed to be sending out broadcast ARP requests. When you do something that provokes an ARP request, you see two or three ARP broadcast packets from the machine in close succession (but not back to back; timestamps say there's a little bit of time between each). That sounds okay, doesn't it? Or at least it's not too crazy. There's a plausible case for rapid repeated ARPs in cases where the first request didn't get an immediate reply, and probably Unixes behave differently here so experience on say Linux doesn't necessarily tell you what to normally expect on OpenBSD or Illumos.

Except that there's a problem here. As far as I know, there is no way to tell from tcpdump output whether you're seeing these packets because the system is transmitting them or because it's receiving them. Of course normally you shouldn't (re-)receive packets that your system initially transmitted, but, well, network loops can happen.

Some versions of tcpdump have a switch to control whether it listens to input packets, output packets, or both. On OpenBSD this is -D, on different versions of Linux it is either -P (Ubuntu, CentOS) or -Q (Fedora); FreeBSD doesn't seem to have an option for this. Of course to use this option (if it's available) you have to remember that some sort of echo-back situation might be happening, but at least you can check for it.

This is definitely something that I'm going to have to try to remember for future network troubleshooting. Sadly it's not as simple as always using 'in' initially, because often you want to see both what the machine is sending and what it's getting back; you just would like to be able to tell them apart immediately.

(I believe that this is a limitation of the underlying kernel interfaces that tcpdump uses, in that most or all implementations simply don't tag packets with whether they're 'out' or 'in' packets.)

(This is just one of a number of ways that I've found to be misled by not looking sufficiently closely at what tcpdump seems to be telling me. Eg tcpdump -p versus not, various firewall and IPSec settings causing received packets to be dropped or even IPSec re-materializing packets, and not looking at MAC addresses (also).)

Comments on this page:

Wouldn't the source MAC address give you some clue as to the where the frame originated from?

By cks at 2016-01-18 02:25:43:

Unfortunately not for a pure network-level loop. In that, the packet you see again is your original packet (with the original MAC), just circulated back inside the switch fabric. The only time the packet's MAC will change is if something touches it at a higher level and re-injects it with its own MAC attached. I think this is fairly unlikely for most broadcast traffic, such as ARP requests.

Dammit. I'm sure I saw this feature in Wireshark, but it's not there now. packet(7) says the kernel does expose this.

sll_pkttype contains the packet type. Valid types are PACKET_HOST for a packet addressed to the local host, PACKET_BROADCAST for a physical-layer broadcast packet, PACKET_MULTICAST for a packet sent to a physical-layer multicast address, PACKET_OTHERHOST for a packet to some other host that has been caught by a device driver in promiscuous mode, and PACKET_OUTGOING for a packet originating from the local host that is looped back to a packet socket. These types make sense only for receiving.

Do the inbound and outbound qualifiers in the pcap filter expression not do what you want? These are not very well documented (I see them mentioned briefly in pcap-filter(7)). I have used them successfully in the past, but I'm not sure how long they have been present in libpcap.

By cks at 2016-01-19 12:23:08:

As far as I know, there's no way to make tcpdump use those qualifiers to print inbound and outbound packets with an explicit marker of which is which. That's what I really want (and in fact I want it to happen by default). Then something like:

> 12:20:43.861401 arp who-has tell
< 12:20:43.972301 arp who-has tell

would be a glaring sign that you are seeing a packet echo, not two fast ARP requests from the host for whatever reason.

Written on 18 January 2016.
« My theory on how network loop caused the problem we observed
Today I learned that a syslog server can be very silent on the network »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jan 18 00:46:47 2016
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.