An irritating OpenBSD PF limitation on redirections

April 4, 2013

I am generally fond of OpenBSD's PF packet filter but every so often I run across a seemingly arbitrary limitation that drives me up the wall. Today's limitation is on where you can redirect packets to as part of NAT'ing and general address translation. I'll start by sketching out a simplified version of the problem I'm trying to solve.

Part of our complex networking setup is a scheme where specific internal machines, sitting on 'sandbox' subnets in private address space, can be reached by the outside world through public IP addresses that sit on what is effectively a virtual subnet. Through a complex dance involving two firewalls, these machines are bidirectionally NAT'd to their public IPs when they talk to the outside world. Our problem is that sometimes internal machines try to use the public IPs, and we'd like to make that work. What we want to do is conceptually simple: when a packet from the internal network and to the public IP shows up on the sandbox firewall, it should be rewritten to the internal IP instead and put back on the internal network. Something like, in pf-ese:

pass in quick on $int_if from <int_lan> to $PUBIP rdr-to $INTIP route-to $int_if

(It's not necessary to rewrite the source address and in fact it's a feature to not do so. Update: as covered in comments, it may be necessary to rewrite the source address to force return traffic to flow through the firewall to be fixed up.)

As it happens, OpenBSD PF is specifically documented (in the pf.conf manpage) to not allow this:

Redirections cannot reflect packets back through the interface they arrive on, they can only be redirected to hosts connected to different interfaces or to the firewall itself.

In the fine OpenBSD tradition this is in fact not completely true. The specific LAN segment that is $int_if actually has two separate subnets on it for historical reasons and machines on the other subnet can talk to $INTIP through this rdr-to rule without problems. It's only machines on the same subnet that can't (and not because PF blocks the packets; I've checked).

What I assume is happening is that PF and OpenBSD's routing stack are interacting badly. Under normal circumstances a router will not route a packet from host A on a subnet to host B on the same subnet (at most it will send an ICMP redirect). In an ideal world PF would be able to bypass this restriction when it rdr-to's something, especially with an explicit route-to (in my books, route-to should mean 'shut up and send the packet out that interface no matter what'). In this world PF apparently can't, which is an irritating limitation that gets in the way of what I maintain is a perfectly sensible thing to want to do.

(There are any number of cases where you might want to redirect traffic nominally to the outside world back to an internal machine.)

PS: as the pf.conf manpage notes, theoretically the way around this is to add NAT'ing with a pass out rule. I was unable to get this to work when I tried it but I might have been using options that were slightly wrong. I assume that this NAT'ing process is enough to fool the routing system into accepting the packet as something that could be validly routed.

(On the other hand, if 'pass out' is applied after routing is done I don't see how this can work. It would make sense for it to be a post-routing action, since routing is what normally decides the outbound interface, but the pf.conf manpage doesn't document whether this is the case or whether some deep magic is happening.)

Comments on this page:

From at 2013-04-04 06:10:51:

It's not necessary to rewrite the source address and in fact it's a feature to not do so.

But then, if Ai host tries to connect to Be address, and Be is rewritten to Bi, Bi replies directly to Ai with Bi as a source of reply. Ai expects a reply from Be and not from Bi. In TCP you would get RST. How would you handle this?

-- dozzie

By cks at 2013-04-04 09:00:49:

You know, you're right. For some reason I was thinking of it like DNS lookups but IP doesn't really work like that. I think it worked for the different-subnet case because the return traffic came through the firewall again and had the transformation reversed. The same-subnet case will clearly require the source address to change via NAT (to the firewall) so that return traffic re-traverses the firewall.

From at 2013-04-17 23:21:07:

I have my redirections working. It's not a great solution, since it requires traffic to bounce through the same interface twice; it's only a placeholder while I have my two-interface machine until I get my multi-interface one and establish a proper DMZ or two (which solves ALL THE PROBLEMS).

It's a bit cumbersome, though, especially if I want my firewall (which also acts as the DNS lookup server, which must also look up from authoritative servers on my LAN) to be able to talk to internal servers when they're pointed at an external IP. That's not going to be a problem for everyone, but I'm lucky enough to have a /29 static IP block, so I'm taking advantage of it.

So for each externally-visible host, I have this set of lines (assuming an earlier global block in on $ext_if line):

# Ports and binat for server 0.
block in on $ext_if from any to $s0_ext
pass in on $ext_if inet proto udp from any to $s0_ext port {$s0_udp}
pass in on $ext_if inet proto tcp from any to $s0_ext port {$s0_tcp}
match on $ext_if from $s0_int to any binat-to $s0_ext
match out on $ext_if from any to $s0_ext rdr-to $s0_int
match in on $int_if from $int_if:network to $s0_ext rdr-to $s0_int

The second-to-last rule is important for things like, "What happens when the router tries to talk to $s0_ext?", which was breaking hostname lookups on locally-hosted DNS until I added the rules.

After that, my LAN NAT rule:

match out on $ext_if from $int_if:network to any nat-to $ext_ip

And finally, the one rule to... ring them all?

# If we've redirected to an internal server, we need to perform some NAT magic
# because the internal server will try to reply directly to the (unmodified)
# source address, leading to some unfortunate confusion.  Essentially, this
# rule says that if we're trying to route from the internal network TO a host
# on the internal network through the router, we need to NAT it or someone will
# be getting replies from an address they're not expecting.
match out on $int_if from $int_if:network to $int_if:network nat-to $int_if

This has worked well for me. I'm eagerly anticipating the arrival of the real hardware I'll be running on, though, because doing all this without a DMZ isn't a great idea and it adds a lot of complexity to the process (and, of course, the aforementioned doubling in traffic for local stuff that's not pointed directly at the local IP).

- Dave Riley (fraveydank at google's mail thing)

Written on 04 April 2013.
« How to make sysadmins unhappy with your project's downloads
Authoritative, non-recursive DNS servers now need ratelimiting »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 4 02:54:24 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.