Something to remember about networking restarts

I helped a coworker debug an interesting Linux problem today, and in the process wound up reinforcing something that I already more or less knew but hadn't really thought about for a while. Namely:

/etc/rc.d/init.d/network restart doesn't unload and reload the kernel modules for your network devices.

A networking restart will bring down and then restart all of your interfaces, reset parameters and so on, but the one thing it doesn't reset is kernel modules. They get loaded only once, the first time around.

Usually this doesn't matter. When it does matter is when you've changed some module parameters (or shuffled what kernel module maps to what ethN number); your changes won't take effect until the next time everything is unloaded and reloaded. This is especially pernicious for virtual network devices, like the bonding driver, where one of the module parameters is how many actual network devices to create.

The bonding driver is what we were scratching our heads over today. My coworker was trying to bring up a second bonding interface on a system that already had one configured, and it just wasn't going; no matter what he did, the bond1 device didn't appear. We added things to /etc/modprobe.conf, we restarted networking until the cows came home, we fiddled things, and nothing worked until we explicitly unloaded the module and reloaded it. We theorize that when things were initially set up, the module parameter that tells the bonding module to create more than one bonding device wasn't set.

(In the process I committed the classical mistake of doing 'network stop' while being logged in over the network. Fortunately this server had one of those 'remote console access' cards so we did not have to make a machine room visit.)