You probably want to start using the
-w option with
The other day, I got notified that my office workstation had an
exposed portmapper service. That was frankly weird, because while
rpcbind running for some NFS experiments, I'd carefully
iptables to block almost all access to it. Or at least I
thought I had; when I looked at '
iptables -vnL INPUT', my blocks
on tcp:111 were conspicuously missing (although it did have the
explicit allow rules for the good traffic). So I went through
systemd's logs from when my own service for installing all of my
IP security rules was starting up and, well:
Sep 22 10:14:30 <host> blocklist: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
I have mixed feelings about this message. On the one hand, it's convenient when programs tell you exactly how they've made your life harder. On the other hand, it's nicer if they don't make your life harder in the first place.
So, the short version of what went wrong is that (modern) Linux
iptables only allows one process to be playing around with iptables
at any given time. If this happens to you, by default
just errors out, printing a helpful message about how it knows what
you probably want to do but it's not going to do it because of
reasons (I'm sure they're good reasons, honest).
(It also applies to
ip6tables, and it appears that
ip6tables share the same lock. The lock is global, not per-chain
or per-table or anything.)
Now, you might think that I was foolishly running two sets of
iptables commands at the same time. It turns out that I probably
was, but it's not obvious, so let's follow along. According to the
logs, the other thing happening at this point during boot was that
my IKE daemon was starting. It was starting
in parallel because this is a Fedora machine, which means systemd,
and systemd likes to do things in parallel whenever it can (which
in practice means whenever you don't prevent it from doing so).
As part of starting up, the Fedora
# Check for nflog setup ExecStartPre=/usr/sbin/ipsec --checknflog
This exists to either set up or disable 'iptables rules for the
nflog devices', and it's implemented in the
ipsec shell script
by running various
iptables commands. Even if you don't have any
nflog settings in your
ipsec.conf and there aren't any devices
ipsec runs at least one
iptables command to verify
this. This takes the lock, which collided with my own IP security
(If you guessed that the
ipsec script does not use '
-w', you win a no-prize. From casual inspection, the script just
assumes that all
iptables commands work all the time, so it isn't
at all prepared for them to fail due to locking problems.)
This particular iptables change seems to have been added in 2013,
in this commit
Either many projects haven't noticed or many projects have the
problem that they need to be portable to
iptables versions that
don't have a
-w argument and so will fail completely if you try
to use '
iptables -w'. I suspect it's a bit of both, honestly.
(Of the supported Linux versions that we still use, Ubuntu 12.04
LTS and RHEL/CentOS 6 don't have '
iptables -w'. Ubuntu 14.04 and
RHEL 7 have it.)
PS: My solution was to serialize IKE IPSec startup so that it was
forced to happen after my IP security stuff had finished; this was
straightforward with a systemd override via '
ipsec.service'. I also went through my own stuff to add '
all of my
iptables invocations, because it can't hurt and it
somewhat protects me against any other instances of this.