Chris's Wiki :: blog/linux/AliasDefaultRouteMistake

How not to set up IP aliases on Ubuntu (and probably Debian)

April 6, 2010

Suppose that you need some IP aliases on an Ubuntu machine. So you go to /etc/network/interfaces and slavishly make yourself some, copying the main stanza a number of times to make entries that looks like this:

auto eth0:0
iface eth0:0 inet static
  address 128.100.1.A
  network 128.100.1.0
  netmask 255.255.255.0
  broadcast 128.100.1.255
  gateway 128.100.1.254

(repeat for every additional IP alias, increasing the number and replacing A with B and so on for all of the different IP aliases.)

What's wrong here is the additional gateway statements for each IP alias; you do not want to specify gateways for IP aliases. The problem with all of these gateway statements is that they create multiple default routes:

$ ip route list | fgrep default default via 128.100.1.254 dev eth0 src 128.100.1.A metric 100 default via 128.100.1.254 dev eth0 src 128.100.1.B metric 100 default via 128.100.1.254 dev eth0 src 128.100.1.C metric 100 default via 128.100.1.254 dev eth0 metric 100

(You have to use 'ip route list' to see this; 'nestat -nr' will tell you that you have multiple default routes but not how they differ.)

These routes differ only in that three of the four specify that the local IP address is something besides the machine's primary IP address (the 'src <IP>' bit).

When you have multiple default routes with the same metric, Linux picks which one to use semi-randomly (and it will change which one it uses from time to time). Since different default routes come with different local IP addresses, your outgoing connections (and UDP requests) will periodically come from a different IP address. This is comedy gold, especially when combined with a cautiously configured firewall that hasn't been configured to pass outbound traffic from some (but not all) of those IP addresses.

Troubleshooting this is part of where the comedy gold comes in; things will work sometimes and not at other times, with the problem coming and going randomly (in reality it comes and goes as the machine chooses different default routes to use, with different local IP addresses). You can have a 'telnet outside-host port' command fail and then your TCP-based traceroute succeed and look fine, for example.

(This happened to us on an Ubuntu 8.04 system. Since Ubuntu and Debian use basically the same system for handling network configuration, I suspect that it would also happen on a Debian machine. It may also happen in other distributions, depending on what they do when you give an IP alias a gateway.)

Comments on this page:

From 114.134.11.1 at 2010-04-06 01:51:30:

I generally prefer to create a dummy interface and bind all my special IP's onto that. It means for instance if one of your nic's goes down (for whatever reason) then those IP's won't vanish with it. It also means that those IP's are /32's and thus don't aquire gateways and so on. It makes firewalling much easier (on the host) as you know what is going on with each interface. There are obvious times when this isn't the right advice.

Although I generally exist in a world where most machines are routers, and participate in a dynamic interior routing protocol of some kind, and thus there are multiple ways to reach the machine any of which must be assumed can go away at anytime.

-- Perry Lorier

From 91.189.88.12 at 2010-04-08 08:32:20:

This is probably what you want:

 auto eth0:0
 iface eth0:0 inet static
   address 128.100.1.A
   netmask 255.255.255.255

By cks at 2010-04-14 13:02:27:

Belatedly, in reply to Perry Lorier:

Binding special IPs to a dummy interface is a nice idea and I may need to play with it someday. We haven't done this in part because essentially all of our machines are single-homed (and none of them are routers), so there's no point in keeping IP alias reachability up if that interface goes down.

From 80.168.55.4 at 2011-08-04 08:27:52:

Red Hat and derivatives check for an identical default route on the parent device in the /etc/sysconfig/network-scripts/ifup-aliases script, and, if it exists, the script does not also set it up on the alias.

However, the script only does a bash string comparison with the config file for the parent device, and does not properly sanitize the gateway address.

So if you have extraneous whitespace or non-printing characters in one or both of your config files, or the default route has been changed manually with the route command, it will still happily call the route command and attempt to setup the duplicate default route.

Written on 06 April 2010.

«	An important thing about how ZFS handles spares

Our current mail system's configuration