Wandering Thoughts archives

How I made IPSec IKE work for a point to point GRE tunnel on Fedora 20

The basic overview of my IPSec needs is that I want to make my home machine (with an outside address) appear as an inside IP address on the same subnet as my work machine is on. Because of Linux proxy ARP limitations, the core mechanics of this involve a GRE tunnel, which must be encrypted and authenticated by IPSec. Previously I was doing this with a static IPSec configuration created by direct use of setkey, which had the drawback that it didn't automatically change encryption keys or notice if something went wrong with the IPSec stuff. The normal solution to these drawbacks is to use an IKE daemon to automatically negotiate IPSec (and time it out if the other end stops), but unfortunately this is not a configuration that IKE daemons such as Fedora 20's Pluto support directly. I can't really blame them; anything involving proxy ARP is at least reasonably peculiar and most sane people either use routing on subnets or NAT the remote machines.

My first step to a working configuration came about after I fixed my configuration to block unprotected GRE traffic. Afterwards I realized this meant that I could completely ignore managing GRE in my IKE configuration and only have it deal with IPSec stuff; I'd just leave the GRE tunnel up all the time and if IPSec was down, the iptables rules would stop traffic. After I gritted my teeth and read through the libreswan ipsec.conf manpage, this turned out to be a reasonably simple configuration. The core of it is this:

conn cksgre
    left=<work IP alias>
    leftsourceip=<work IP alias>
    right=<home public IP>
    ikev2=insist
    # what you want for always-up IPSec
    auto=start
    # I only want to use IPSec on GRE traffic
    leftprotoport=gre
    rightprotoport=gre

    # authentication is:
    authby=rsasig
    rightrsasigkey=[...]
    leftrsaksigkey=[...]

The two IP addresses used here are the two endpoints of my GRE tunnel (the 'remote' and 'local' addresses in 'ip tunnel <...>'). Note that this configuration has absolutely no reference to the local and peer IP addresses that you set on the inside of the tunnel; in my setup IPSec is completely indifferent to them.

I initially attempted to do authentication via PSK aka a (pre) shared secret. This caused my setup of the Fedora 20 version of Pluto to dump core with an assertion failure (for what seems to be somewhat coincidental reasons), which turned out to be lucky because there's a better way. Pluto supports what it calls 'RSA signature authentication', which people who use SSH also know as 'public key authentication'; just as with SSH, you give each end its own keypair and then list the public key(s) in your configuration and you're done. How to create the necessary RSA keypairs and set everything up is not well documented in the Fedora 20 manpages; in fact, I didn't even realize it was possible. Fortunately I stumbled over this invaluable blog entry on setting up a basic IPSec connection which covered the magic required.

This got the basic setup working, but after a while the novelty wore off and my urge to fiddle with things got the better of me so I decided to link the GRE tunnel to the IKE connection, so it would be torn down if the connection died (and brought up when the connection was made). You get your commands run on such connection events through the leftupdown="..." or rightupdown="..." configuration setting; your command gets information about what's going on through a pile of environment variables (which are documented in the ipsec_pluto manpage). For me this is a script that inspects $PLUTO_VERB to find out what's going on and runs one of my existing scripts to set up or tear down things on up-host and down-host actions. As far as I can tell, my configuration does not need to run the default 'ipsec _updown' command.

(My existing scripts used to do both GRE setup and IPSec setup, but of course now they only do the GRE setup and the IPSec stuff is commented out.)

This left IPSec connection initiation (and termination) itself. On my home machine I used to bring up and tear down the entire IPSec and GRE stuff when my PPPoE DSL link came up or went down. In theory one could now leave this up to a running Pluto based on its normal keying retries and timeouts; in practice this doesn't really work well and I wound up needing to do manual steps. Manual control of Pluto is done through 'ipsec whack' and if everything is running smoothly doing the following on DSL link up or down is enough:

ipsec whack --initiate|--terminate --name cksgre >/dev/null 2>&1

Unfortunately this is not always sufficient. Pluto does not notice dynamically appearing and disappearing network links and addresses, so if it's (re)started while my DSL link is down (for example on boot) it can't find either IP address associated with the cksgre connection and then refuses to try to do anything even if you explicitly ask it to initiate the connection. To make Pluto re-check the system's IP addresses and thus become willing to activate the IPSec connection, I need to do:

ipsec whack --listen

Even though the IPSec connection is set to autostart, Pluto does not actually autostart it when --listen causes it to notice that the necessary IP address now exists; instead I have to explicitly initiate it with 'ipsec whack --initiate --name cksgre'. My current setup wraps this all up in a script and runs it from /etc/ppp/ip-up.local and ip-down.local (in the same place where I previously invoked my own IPSec and GRE setup and stop scripts).

So far merely poking Pluto with --listen has been sufficient to get it to behave, but I haven't extensively tested this. My script currently has a fallback that will do a 'systemctl restart ipsec' if nothing else works.

PS: Note that taking down the GRE tunnel on IPSec failure has some potential security implications in my environment. I think I'm okay with them, but that's really something for another entry.

Sidebar: What ipsec _updown is and does

On Fedora 20 this is /usr/libexec/ipsec/_updown, which runs one of the _updown.* scripts in that directory depending on what the kernel protocol is; on most Linux machines (and certainly on Fedora 20) this is NETKEY, so _updown.netkey is what gets run in the end. What these scripts can do for you and maybe do do for you is neither clear nor documented and they make me nervous. They certainly seem to have the potential to do any number of things, some of them interesting and some of them alarming.

Having now scanned _updown.netkey, it appears that the only thing it might possibly be doing for me is mangling my /etc/resolv.conf. So, uh, no thanks.

Using iptables to block traffic that's not protected by IPSec

When I talk about my IPSec setup, I often say that I use GRE over IPSec (or 'an IPSec based GRE tunnel'). However, this is not really what is going on; a more accurate but more opaque description is that I have a GRE tunnel that is encrypted and protected by IPSec. The problem, and the reason that the difference matters, is that there is nothing that intrinsically ties the two pieces together, unlike something where you are genuinely running X over Y such as 'forwarding X11 over SSH'. In the X11 over SSH case, if SSH is not working you do not get anything. But in my case if IPSec isn't there for some reason my GRE tunnel will cheerfully continue working, just without any protection against either eavesdropping or impersonation.

In theory this is undoubtedly not supposed to happen, since you (I) designed your GRE setup to work in conjunction with IPSec. Unfortunately in practice in practice there are any number of ways for IPSec to go away on you, possibly without destroying the GRE tunnel in the process. Your IPSec IKE daemon probably removes the IPSec security policies that reject unencrypted traffic when it shuts down, for example, and if you're manually configuring IPSec with setkey you can do all sorts of fun things like accidentally leaving a 'spdflush;' command in a control file that only (re)loads keys and is no longer used to set up the security policies.

The obvious safety method is to add some iptables rules that block unencrypted GRE traffic. If you are like me, you'll start out by writing the obvious iptables ruleset:

iptables -A INPUT -p esp -j ACCEPT
iptables -A INPUT -p gre -j DROP

This doesn't work. As far as I can tell, the Linux IPSec system effectively re-injects the decrypted packets into the IP stack, where they will be seen in their unencrypted state by iptables rules (as well as by tcpdump, which can be both confusing and alarming). The result is that after the re-injection the ipfilters rules see a plain GRE packet and drop it.

Courtesy of this netfilter mailing list message, it turns out that what you need is to match packets that will be or have been processed by IPSec. This is done with a policy match:

iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT
iptables -A INPUT -p gre -j DROP

# and for outgoing packets:
iptables -A OUTPUT -m policy --dir out --pol ipsec -j ACCEPT
iptables -A OUTPUT -p gre -j DROP

Reading the iptables-extensions manpage suggests that I should add at least '--proto esp' to the policy match for extra paranoia.

I've tested these rules and they work. They pass GRE traffic that is protected by IPSec, but if I remove the IPSec security policies that force IPSec for my GRE traffic these iptables rules block the unprotected GRE traffic as I want.

(Extension to non-GRE traffic is left as an exercise to the reader. I have a simple IPSec story in that I'm only using it to protect GRE and I never want GRE traffic to flow without IPSec to any destination under any circumstances. Note that there are potentially tricky rule ordering issues here and you probably want to always put this set of rules at the end of your processing.)

My Linux IPSec/VPN setup and requirements

In response to my entry mentioning perhaps writing my own daemon to rekey my IPSec tunnel, a number of people made suggestions in comments. Rather than write a long response, I've decided to write up how my current IPSec tunnel works and what my requirements are for it or any equivalent. As far as I know these requirements rule out most VPN software, at least in its normal setup.

My IPSec based GRE tunnel runs between my home machine and my work machine and its fundamental purpose is to cause my home machine to appear on the work network as just another distinct host with its own IP address. Importantly this IP address is publicly visible, not just an internal one. My home machine routes some but not all of its traffic over the IPSec tunnel and for various reasons I need full dual identity routing for it; traffic to or from the internal IP must flow over the IPSec tunnel while traffic to or from the external IP must not. My work machine also has additional interfaces that I need to isolate, which can get a bit complicated.

(The actual setup of this turns out to be kind of tangled, with some side effects.)

This tunnel is normally up all of the time, although under special circumstances it needs to be pulled down locally on my work machine (and independently on my home machine). Both home and work machines have static IPs. All of this works today; the only thing that my IPSec setup lacks is periodic automatic rekeying of the IPSec symmetric keys used for encryption and authentication.

Most VPN software that I've seen wants to either masquerade your traffic as coming from the VPN IP itself or to make clients appear on a (virtual) subnet behind the VPN server with explicit routing. Neither is what I want. Some VPNs will bridge networks together; this is not appropriate either because I have no desire to funnel all of the broadcast traffic running around on the work subnet over my DSL PPPoE link. Nor can I use pure IPSec alone, due to a Linux proxy ARP limitation (unless this has been fixed since then).

I suspect that there is no way to tell IKE daemons 'I don't need you to set things up, just to rekey this periodically'; this would be the minimally intrusive change. There is probably a way to configure a pair of IKE daemons to do everything, so that they fully control the whole IPSec and GRE tunnel setup; there is probably even a way to tell them to kick off the setup of policy based routing when a connection is negotiated. However for obvious reasons my motivation for learning enough about IKE configuration to recreate my whole setup is somewhat low, as much of the work is pure overhead that's required just to get me to where I already am now. On the other hand, if a working IKE based configuration for all of this fell out of the sky I would probably be perfectly happy to use it; I'm not intrinsically opposed to IKE, just far from convinced that investing a bunch of effort into decoding how I need to set it up will get me much or be interesting.

(It would be different if documentation for IKE daemons was clear and easy to follow, but so far I haven't found any that is. Any time I skim any of it I can see a great deal of annoyance in my future.)

PS: It's possible that my hardcoded IPSec setup is not the most modern in terms of security, since it dates from many years ago. Switching to a fully IKE-mediated setup would in theory give me a free ride on future best practices for IPSec algorithm choices so I don't have to worry about this.

Sidebar: why I feel that writing my own rekeying daemon is safe

The short version is that the daemon would not involved in setting up the secure tunnel itself, just getting new keys from /dev/urandom, telling the authenticated other end about them, writing them to a setkey script file, and running the necessary commands to (re)load them. I'd completely agree with everyone who is telling me to use IKE if I was attempting to actively negotiate a full IPSec setup, but I'm not. The IPSec setup is very firmly fixed; the only thing that varies is the keys. There are ways to lose badly here, but they're almost entirely covered by using a transport protocol with strong encryption and authentication and then insisting on fixed IP addresses on top of it.

(Note that I won't be negotiating keys with the other end as such. Whichever end initiates a rekey will contact the other end to say more or less 'here are my new keys, now use them'. And I don't intend to give the daemon the ability to report on the current keys. If I need to know them I can go look outside of the daemon. If the keys are out of sync or broken, well, the easy thing is to force an immediate rekey to fix it, not to read out current keys to try to resync each end.)

Hassles with getting our NFS mount authentication working on Linux

Our existing Solaris NFS fileservers have a custom NFS mount authentication method to do relatively strong authentication of a machine's identity before we allow it to establish a NFS mount from us. For various reasons we've started looking at doing NFS service from Linux machines and so we need to implement some version of our NFS mount authentication for them (ideally one that looks exactly the same from the client side).

Our existing Solaris mechanism uses a NSS netgroup module that does the authentication as part of checking netgroup membership. Given that Linux has NSS modules and /etc/nsswitch.conf and so on, I figured that this would be the easiest and most natural way to implement our system on Linux. Unfortunately this ran into the roadblock that you can't write a NSS module that implements netgroups without using internal glibc headers that define an internal glibc struct that your functions must manipulate. In effect you can't write NSS netgroup modules at all.

(Since this is an internal header glibc is free to change the structure at any time and thereby make your existing compiled module silently binary incompatible.)

This leaves me trying to figure out the best alternate approach. Right now I can think of three:

• Modify mountd itself, since we have the source code. The two drawbacks to this is that we have to figure out where (and how) to modify the mountd source and then we have to keep our hack up to date as new versions of mountd are released. This implies we'd be replacing a standard Ubuntu or CentOS package with a local version, which doesn't make me really happy.

• Switch to IPSec for clients that need NFS mount authentication. The obvious drawbacks are that this is likely to be (much) slower and we have to get IPSec working, probably in its more complex form with IKE.

• Use a firewall based mechanism to block access to the server's NFS and mountd daemons until a client has been authenticated (this might be a good use for ipsets). The drawback of this approach is that it requires the most custom software on the server and probably the client. I'd like some way to intercept mount requests, trigger our actions, and then let the mount request go on, but I don't know if that's even possible (or at least possible without deep netfilter hacks).

I plan to ask the linux-nfs mailing list if they have any clever ideas that I'm missing or any opinions on the best approach to do this, although I suspect that they'll tell me to use IPSec (it's the obvious solution apart from the performance hit).

The whole situation with glibc mis-managing NSS netgroup support irritates me. NSS is theoretically supposed to enable people to add their own modules but glibc has crippled this part of it for no good reason and made our life significantly more painful in the process. I'd report this as a bug to the glibc people except that the idea of reporting any 'features missing or misimplemented' issue to glibc is utterly laughable; not only would my problems not get solved any time soon even in the best case, but historically it has been an excellent way to get abused.

(While there are NSS modules outside of glibc I don't believe that any of them support netgroups, just things like users and groups and passwords. Not that glibc even really documents how to write those sorts of NSS modules either.)

What I'm worried about with retina displays on Linux

I've been waiting for high-DPI LCD panels to appear for years, so I'm both happy and excited to see 'retina' displays start to appear on desktop machines. Recent announcements even put a usable such panel within theoretical reach. But this means that I have to start worrying about one of the problems I may have with them, namely whether or not I'll be able to drive such a high-resolution panel under Linux.

Mac and Windows people don't really have any worries here. Since these panels are useless without the ability to drive them, there's going to be graphics cards that can do the job. By the time the market drives the panel prices down to less stratospheric levels, these cards should even be fairly widely available (although they may well still be relatively high end cards).

But I use Linux and I go further than that by insisting on using open source graphics drivers. The open source drivers almost always lag behind in both hardware support and feature support, while driving really high resolutions is likely to be a leading edge feature for quite a while. I can thus easily imagine a future where the only cards that can drive the high-DPI displays I want are high end ones that aren't supported in open source drivers and won't be for years to come.

(In fact the hardware that's best supported in open source drivers today seems to be Intel's, and who knows if Intel will even make a graphics chipset that can drive big high-DPI displays. So far all Intel chipsets have been integrated into motherboards, and people who buy systems with integrated graphics are often not seen as people with demanding graphics needs.)

By the way, there are of course a lot of software issues that are going to come up with really high DPI displays on Linux; many things work in absolute pixels (which are going to be tiny) and many graphics and icons won't be available large and won't scale up. I'm sure I'll be looking various tiny little pictures for years. But I also expect that all of those issues will get solved relatively fast, partly because they're also an issue on laptops with high-DPI displays.

(I suspect that some of them have already been solved in some desktop environments.)

PS: The current lot of 4K TVs are not useful for me as displays because they have too low a refresh rate. I've historically been quite sensitive to this issue with early LCD panels, so I'm basically sure that 30 Hz is not enough for my usage. Future 4K TV panels will probably improve that, but I'm holding out for high-DPI 5K displays since they've started to show up too.