Wandering Thoughts archives

You probably want to start using the -w option with iptables

The other day, I got notified that my office workstation had an exposed portmapper service. That was frankly weird, because while I had rpcbind running for some NFS experiments, I'd carefully used iptables to block almost all access to it. Or at least I thought I had; when I looked at 'iptables -vnL INPUT', my blocks on tcp:111 were conspicuously missing (although it did have the explicit allow rules for the good traffic). So I went through systemd's logs from when my own service for installing all of my IP security rules was starting up and, well:

Sep 22 10:14:30 <host> blocklist[1834]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

I have mixed feelings about this message. On the one hand, it's convenient when programs tell you exactly how they've made your life harder. On the other hand, it's nicer if they don't make your life harder in the first place.

So, the short version of what went wrong is that (modern) Linux iptables only allows one process to be playing around with iptables at any given time. If this happens to you, by default iptables just errors out, printing a helpful message about how it knows what you probably want to do but it's not going to do it because of reasons (I'm sure they're good reasons, honest).

(It also applies to ip6tables, and it appears that iptables and ip6tables share the same lock. The lock is global, not per-chain or per-table or anything.)

Now, you might think that I was foolishly running two sets of iptables commands at the same time. It turns out that I probably was, but it's not obvious, so let's follow along. According to the logs, the other thing happening at this point during boot was that my IKE daemon was starting. It was starting in parallel because this is a Fedora machine, which means systemd, and systemd likes to do things in parallel whenever it can (which in practice means whenever you don't prevent it from doing so). As part of starting up, the Fedora ipsec.service has:

# Check for nflog setup
ExecStartPre=/usr/sbin/ipsec --checknflog

This exists to either set up or disable 'iptables rules for the nflog devices', and it's implemented in the ipsec shell script by running various iptables commands. Even if you don't have any nflog settings in your ipsec.conf and there aren't any devices configured, ipsec runs at least one iptables command to verify this. This takes the lock, which collided with my own IP security setup scripts.

(If you guessed that the ipsec script does not use 'iptables -w', you win a no-prize. From casual inspection, the script just assumes that all iptables commands work all the time, so it isn't at all prepared for them to fail due to locking problems.)

This particular iptables change seems to have been added in 2013, in this commit (via). Either many projects haven't noticed or many projects have the problem that they need to be portable to iptables versions that don't have a -w argument and so will fail completely if you try to use 'iptables -w'. I suspect it's a bit of both, honestly.

(Of the supported Linux versions that we still use, Ubuntu 12.04 LTS and RHEL/CentOS 6 don't have 'iptables -w'. Ubuntu 14.04 and RHEL 7 have it.)

PS: My solution was to serialize IKE IPSec startup so that it was forced to happen after my IP security stuff had finished; this was straightforward with a systemd override via 'systemctl edit ipsec.service'. I also went through my own stuff to add '-w' to all of my iptables invocations, because it can't hurt and it somewhat protects me against any other instances of this.

Git's selective commits plus Magit are a killer feature for me

I'm sure there are some people who are meticulously organized in their programming work. They work on only one thing at a time, or if they're making multiple changes they carefully separate them on different topic branches. I'm not one of them. I'm working away on something, but then I can't resist improving something that I stumble over because I was in that area of the code, and I run into a bug that needs to be corrected, and by the time I turn around there's a bunch of unrelated changes all piled in together.

I think git has had 'git add -p' for as long as I've been using it, but in practice it was never usable enough for me. I couldn't stand the tedious grind needed and I made mistakes and sometimes the changes I wanted to separated were in what git considered one chunk. I made a couple of dutiful attempts to use it and make those proper neat commits but soon gave it up as too much of a pain. If I was lucky my unrelated changes were in separate files and I'd make multiple commits; otherwise, well, there were big commits with big change lists and sometimes casual admissions in asides that I'd also done some additional work.

When I first started with Magit I didn't really expect anything much to change with me and git. Sure, I was picking Magit up to make selective commits easier and it did that, but I didn't think that was all that big. After all, most of my separate changes were already to separate files; I at least remembered the really entangled changes as rare.

I was wrong. Easy selective commits have turned into a killer feature of the git plus Magit combination, and I've wound up making them all the time. I think part of it is that having them available is by itself liberating, and encourages me to make most changes freely. I know that I can sort everything out later, that I can let changes mature and evolve at different rates, and so on. So any time I see something I want to tweak or fix or correct, I can do it right then and there. So I do.

(This is especially useful for small single-file programs, such as this one I've been working on recently. Almost all of the commits to it are entangled ones that I sorted out with Magit.)

Sidebar: Magit's taking over making almost all of my commits

Why is pretty straightforward; it's a nicer environment than moving between multiple xterm windows to check diffs, stage changes, make a commit, call up an editor and a spell checker, and so on. Magit has conveniences like warning me if the first line of the commit is getting too long and it inherits all of the straightforward GNU Emacs features like on the fly spellchecking with flyspell mode (once I looked up and worked out how to add it to my Emacs setup in a way that worked). So I wind up writing better commit messages (or at least better spelled ones) with generally less work.

As someone who thinks of himself as a Unix person, I'm not entirely copacetic about GNU Emacs swallowing more of my command line work. But these days I'm a pragmatist too, and Magit and GNU Emacs sure are convenient here. I've even wound up using Magit for some commit amending just because it was easy enough (and easier to look up in the Magit manual than wrestle with figuring out the exact command line incantation I was going to need).