Wandering Thoughts archives


How to quiesce NFS traffic the brute force way

Once upon a time, we had an NFS server. Like sensible people, we immediately put the NFS server on a UPS, which meant that if we lost power, we wanted to quiesce the NFS traffic before we had to shut it down.

There are probably a lot of sophisticated solutions to this problem, involving things like hooking into the UPS monitoring system and triggering events on power loss. We opted for a much simpler method: we didn't put the Ethernet switch that the server was connected to on a UPS.

No power, no switch, no network traffic reaching the server, and it gets to settle down in peace before it has to shut down.

(You might wonder how anyone else could be talking to the server if we'd lost power. The answer is 'different power circuits'; we had machines in other locations, and there was no guarantee that our connection to the campus backbone would be cut if we lost power in our machine room but kept our core switch up.)

In hindsight we should have used two switches, one for the core servers and one for the outside links. That would have let the fileserver talk to the mail server to at least queue up plaintive power loss notices when we actually lost power. (Fortunately, the UPS monitoring software sent those out with a separate process, so the unavailability of email didn't stop it from doing orderly shutdowns as the UPS batteries ran down.)

Looked at from the right angle, we were using our core switch as a fuse. I suspect that there are other applications for this sort of trick if you look for them.

sysadmin/BruteForceQuiesce written at 21:48:57; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.