Wandering Thoughts archives

What promiscuous mode does on modern networks

Recently, I have wound up wondering if using or not using tcpdump's -p switch made any difference on modern hardware and modern switched networks (partly because not using it causes your kernel to generate a message every time you start and stop tcpdump, and I can live without that). The somewhat unfortunate answer is that yes, it still makes a difference.

First, promiscuous mode is still at least partly a low level hardware thing in your network card, not just a software switch inside the networking stack. (The exception is that I believe that some hardware has limited support for multicast, so effectively turns on full promiscuous mode any time you do enough multicast stuff.)

Second, even on normal ports on modern switched networks you can still see traffic that requires promiscuous mode to receive, ie traffic that is neither broadcast nor directed to your machine specifically. Often this is a sign that something weird is going on, which makes it just the sort of thing that you most want to see.

(And of course if you are tapping the network deliberately, for example on a mirror port on a switch, you are sure to receive such traffic.)

I'll still use -p most of the time (and I wish it was the default), because most of the time that I use tcpdump I'm only interested in traffic that is supposed to be flowing through the machine in the first place.

(The kernel itself will pass to tcpdump all packets that it receives from the network card, so if you have a bridged virtual machine and use tcpdump -p on the host machine you will still see traffic to and from the bridged machine.)

Combining dual identity routing and isolated interfaces

In theory, combining dual identity routing with isolated interfaces should be simple. However, it turns out that there is a small trick to it.

This came up today because I am in the process of migrating my office machine from one network to another. This means that I currently have three active interfaces on the machine (old IP address, new IP address, and an internal network), plus one end of a GRE tunnel, and as part of the migration I wanted to force the routing to work right, just in case.

The problem with the straightforward combination is that it will try to force locally generated packets from one of the machine's IP addresses out the appropriate network interface even if it is talking to the other end of the GRE tunnel. This is unlikely to work very well; at the best it will bounce off the gateway router and come right back.

The solution is to steal packets for the GRE tunnel before they get handled by the source IP address rules. Usefully, the isolated interfaces part of this has already defined a routing table with exactly the right IP addresses in it. So we write the ip rule statement as:

# table 10 is from the isolated interfaces
ip rule add iif lo priority 4999 table 10

ip route add default via GW1 dev eth0 table 20
ip rule add from IP1 iif lo priority 5000 table 20

ip route add default via GW2 dev eth1 table 21
ip rule add from IP2 iif lo priority 5001 table 21

ip route add default via GW3 dev eth2 table 22
ip rule add from IP3 iif lo priority 5002 table 22

As a useful side effect, this makes it possible for me to fully test the firewall rules protecting the internal network, since now my machine is willing to actually route packets through the firewall.

(I am tempted to package the ip route/ip rule bits up as a script, just so I can conveniently bring up scratch IP addresses on various VLANs for exactly this sort of firewall testing. The only tricky bit is automatically picking the table number, as ip rule seems happy to have a bunch of rules with the same priority.)

Alas, I am now left with a mystery: according to the policy routing rules, it looks like a packet from IP1 to an address on that subnet should get routed via the gateway (and similarly for the other networks), as the main routing table that has the local-network routing information is only looked at by a rule with priority 32766, much lower than the priority 5000 rule for locally generated packets with source addresses of IP1.

(I just checked and the explicit rule to steal GRE tunnel traffic is necessary; without it the packets get shoved out on eth0 instead.)

Important update, November 20th: it turns out that I was wrong and packets from IP1 to the subnet were getting routed via the gateway. See DualIdentityIsolationII for the fix.

Getting live network bandwidth numbers on Linux

Today I got curious about a simple question: was my iSCSI target machine actually running at its full potential read speed?

The machine exports individual disks to its clients, so measuring single disk performance wouldn't give me the answer. Summing up IO across all the disks would have given me a number, but so would just getting the network bandwidth utilization; if the machine was saturating its gigabit link, it was clearly running as fast as it could.

There doesn't seem to be a program that will directly show this information (at least not on Red Hat Enterprise 5), but you can get the total byte counts for an interface from ifconfig, which means that with a small script I had what I wanted. (Then I rewrote it to read the stats directly from /proc/net/dev instead of running ifconfig and groping through the output.)

Since it may be useful for other people, here's what I'm calling netvolmon:

#!/bin/sh
# usage: netvolmon DEV [INTERVAL]
DEV=$1
IVAL=5
if [ "$#" -eq 2 ]; then
    IVAL=$2
fi

getrxtx() {
    grep "$1:" /proc/net/dev | sed 's/^.*://' |
        awk '{print $1, $9}'
}

rxtx=$(getrxtx $DEV)
while sleep $IVAL; do
    nrxtx=$(getrxtx $DEV)
    (echo $IVAL $rxtx $nrxtx) |
    awk '{rxd = ($4 - $2) / (1024*1024*$1);
          txd = ($5 - $3) / (1024*1024*$1);
          printf "%6.2f MB/s RX %6.2f MB/s TX\n",
                 rxd, txd}'
    rxtx="$nrxtx"
done

Unfortunately this illustrates one reason why shell scripting is so pervasive: it is such a convenient way of banging rocks together in a hurry. Once I hit on the trick of using awk for all the arithmetic, it probably took me longer to fiddle with the output formatting than to write the rest of the script.

(And I have to give bash a big raspberry for making array variables useless for precisely the situation where they would be most useful, namely picking individual elements out of the output of a command that prints multiple pieces of information.)