Wandering Thoughts archives

What Linux's getaddrinfo() is doing with IPv6 (on some versions)

I've previously noted that Linux's getaddrinfo() implementation doesn't return IPv6 addresses by default if it doesn't think that your system is IPv6 capable, or perhaps if your system has IPv6 disabled (this is most conveniently visible through the Python interface). This behavior is more or less the same as what you'd get if you supplied the AI_ADDRCONFIG flag in the hints structure.

(To be specific, I have seen this behavior on Ubuntu 8.04. It doesn't seem to be present in Ubuntu 6.06, and unfortunately I don't have any other Linux machines with IPv6 disabled to test on.)

This behavior is half-documented in modern getaddrinfo() manpages, which say that if you supply no hints structure you get flags that include AI_ADDRCONFIG. However this is not the full story, since you also get this behavior if you supply a hints structure with ai_family set to 0 (aka AF_UNSPEC).

I would love to report that I had found the code that does this in glibc. However, the glibc getaddrinfo() internals have defeated me; I cannot see any clear code that has this particular side effect. So all I can do is report what I have determined experimentally, which is that supplying a non-zero set of flags in af_flags doesn't make a difference, nor does the type of socket you ask for. The only thing that will get you IPv6 addresses on such a Linux system is putting AF_INET6 in the ai_family field, at which point they will magically appear.

(Note that this behavior is subtly different from turning on the AI_ADDRCONFIG flag all the time, as the flag causes IPv6 results to never get returned, even if you ask for them specifically.)

I assume that this is why 'getents ahostsv6' (from the comments here) will fail on such a system; I expect it is making a plain getaddrinfo() call and filtering the results if you ask specifically for IPv6 addresses.

(The getents behavior may be a bug, but the glibc getaddrinfo() behavior is probably deliberate.)

A Linux gotcha about daemons and bindv6only

First, the brief review. Linux's net.ipv6.bindv6only sysctl controls whether an IPv6 socket can also accept IPv4 traffic (with IPv4 mapped addresses), or whether it binds only to true IPv6 traffic. So if you want to turn off dual binding (which I think you should), you turn this sysctl on and then run around fixing up things to work.

Except there's a gotcha. Whether IPv6 sockets can also talk IPv4 is actually a per-socket property, and setting the bindv6only sysctl only sets the default value for new sockets. Programs can override this, as Apache does, and existing server sockets keep their current behavior.

(I expect that sockets created by accept() inherit this property from the listening socket, since that's basically the only sensible way to handle this.)

The net result is that if you enable bindv6only on an already running system, you can get various sorts of misleading and peculiar results. The big misleading result is that any running daemon with a bound IPv6 socket will continue to get connections from IPv4 machines and can probably still talk to them; this will make it look like your system's configuration is more single-bind-ready than it actually is, since the same daemon won't be working so well after a reboot.

The peculiar result is that daemons that sometimes open new connections will probably fail badly. When talking over their regular server socket they will have no problem since that is still dual-bound, but when they go to open a new connection they will fail; they'll create an IPv6 socket (because that matches both their server socket and the type of address they want to talk to) but it will reject their attempts to talk to the IPv4 address.

(I am pretty sure that this is what I saw with the Amanda client setup on one machine.)

The moral is that if you turn on bindv6only, you should immediately hunt down all programs with listening IPv6 sockets and fix any of them that need to talk to IPv4 machines (except for Apache, it handles this on its own). Don't assume that everything is fine just because things seem to still work; they may be subtly broken, and they may be fine only until you reboot.

IPv6 with Lighttpd on Linux

The situation with lighttpd is sufficiently complicated that I can't call it a brief note, so here's what I've worked out. This is not necessarily valid for all distributions, since I'm working on Fedora 11.

Lighttpd is fully IPv6 enabled, but normally only binds its listening socket to the IPv4 address. Some distributions override this by setting the server.use-ipv6 configuration option if IPv6 is enabled on the machine.

Now, server.use-ipv6 is somewhat underdocumented; the lighttpd documentation says only 'bind to the IPv6 socket'. What it actually does at the moment is make lighttpd always create IPv6 sockets, regardless of what the socket is supposed to be bound to. (I am not sure how well this works if you are binding to specific IPv4 addresses, even on a dual binding machine with bindv6only turned off.)

Since lighttpd fully supports IPv6, you do not need to turn server.use-ipv6 on in order to bind to an IPv6 socket; instead, you simply tell lighttpd to do so by giving it an IPv6 address as server.bind or whatever. So if you have turned bindv6only on to turn off dual-binding and thus need to bind separately to both the IPv4 and the IPv6 sockets, you just need to tell lighttpd to do this by adding the following configuration bit:

$SERVER["socket"] == "[::]:80" { }

(As the lighttpd documentation dryly notes, 'note that it's perfectly valid to leave the body of the [$SERVER] conditional empty'. Lighttpd configuration is sometimes full of these magic side effects.)

This is sadly a lot more complicated than the Apache way, which can be described as 'it all just works, you don't have to worry about it' (cf the brief note).

Brief notes on IPv6 support in some Linux programs

Lately, I've been trying to actually use IPv6. This is a good way to find out various annoying things about how well programs support it, and so I'm going to write down some notes about what I've found out so far.

• tcpwrappers fully supports IPv6 based access restrictions, but not as nicely as it does IPv4 ones, and subnet-based restrictions have a less convenient notation. This is documented in the hosts.allow manpage.

  (And wow, have modern versions of tcpwrappers picked up some useful features.)

• ssh supports using IPv6 addresses in from="..." restrictions in authorized_keys files, but it doesn't document this and the degree of support varies between versions. In early versions you can only list full IPv6 addresses (written without '[...]' around them, unlike tcpwrappers); later versions also allow you to use the subnet prefix notation, but inconveniently require the subnet to be 'proper', having a host portion that is all zeros.

  (This is the difference between writing '2002:8064:0333::1/48', which is how a lot of the actual IPv6 stuff is configured, and having to write '2002:8064:0333::/48'. Real examples may be more complicated.)

  Fedora 8 has an earlier ssh; Fedora 11 has a later one.

• xinetd binds to IPv6 sockets by default, which is a problem if you are turning off dual-binding, which is what I think you should do. To specifically bind to IPv4 sockets, set 'flags = ipv4' for a particular entry.

• Apache defaults to listening on an IPv6 socket. You might think that this is a problem if you turn off dual-binding, but in fact Apache is smarter than you; even if you turn off dual-binding, Apache will turn it back on for its server socket.

  (In fact Apache is so smart that if you carefully tell it to listen separately on IPv4 and IPv6 for the same port, it will combine them into one IPv6 socket. This can be really confusing if you've turned off dual-binding and are trying to make everything work and are suddenly doubting whether your configuration file change took because lsof (or netstat) says that Apache only has a single IPv6 socket.)

Apache's dual-binding support is almost completely perfect; if more programs were like it, dual-binding would have a better reputation. Notice that despite it using an IPv6 socket, you never see IPv4 mapped addresses in your logs and you never have to think about them in your configuration files; you just use and see IPv4 addresses, and Apache fixes everything up behind the scenes.

(I dock Apache style points because it does treat IPv4 compatible addresses differently from plain IPv4 addresses. This is defensible, since after all they're using IPv6 to talk to you, but I'm not sure it's desirable.)

Some stuff on dual-bound IPv6 sockets on Linux

One place where IPv4 mapped addresses show up a lot is the logs of Linux daemons that are running on machines with IPv6 enabled. These days, it's reasonably popular for daemons to listen (only) on an IPv6 socket for both IPv4 and IPv6 connections. The kernel makes this work, using IPv4 mapped addresses for the IPv4 connections, and then you get to find out which daemons choke anyways.

Not everyone likes the behavior where listening on a wildcard IPv6 socket will get you both IPv6 and IPv4 connections; see here for a rundown of some of the problems with this. Naturally, there are objections to any change in the status quo.

Personally, I agree with the people arguing against dual-bound sockets who want to set the net.ipv6.bindv6only sysctl to 1 by default. The idea of programs only having to deal with IPv6 is a nice one, but in practice that ship sailed at least a decade ago, and it's too late now. We have a huge collection of configurations and practices with IPv4 addresses; changing them to use the IPv6 form of those addresses is a pointless pain in the rear at best.

(At worst, things don't work as well as they did with real IPv4 addresses. Compare tcpwrappers' support for IPv6 addresses against its support for IPv4 addresses, for example, and consider the work involved in moving an IPv4 tcpwrappers configuration to use IPv4 mapped addresses.)

In practice it's much simpler to keep running IPv4 setups as is and to treat IPv6 as a completely separate world. But in order to make this work you need to be able to listen on a wildcard IPv4 socket and a separate wildcard IPv6 socket at the same time, and to do this you need to turn on bindv6only.

(This has the effect of disabling IPv4 mapped addresses entirely; attempts to use them are rejected by the kernel.)

Having now looked into this issue (my attention was drawn to it by James's comment on this entry), I've now set bindv6only on my machines. It's not proven particularly annoying, as I'm starting from a situation where I have basically nothing binding to IPv6 listening sockets anyways so the setting doesn't actually affect anything.

(Now I do have to start selectively enabling daemons on IPv6, but I probably want to do that anyways in case I need to adjust their configurations. My experience so far is that this is going to be an adventure.)

Beware of using Linux's hostname -s switch

The hostname program has a common switch, -s, which is documented (in the Linux version) as:

-s, --short

Display the short host name. This is the host name cut at the first dot.

Although you would not expect it from this description, running 'hostname -s' will do a gethostbyname() and thus often a DNS lookup in most Linux versions of hostname. This can of course fail if your DNS is not working, which winds up with the very peculiar result of hostname failing. And all of this because you innocently decided to trim out any dots that might be present using the most obvious and easiest approach.

(Most scripts don't cope very well with this, partly because the Bourne shell makes it annoyingly difficult to deal with programs failing in command substitutions and partly because come on, who expects hostname to fail?)

Red Hat Enterprise 5, Fedora 8, Ubuntu 6.06 and Ubuntu 8.04 have versions of hostname that behave this way. Fedora 11 has a version that does not, because someone filed a bug about it; unfortunately I can't tell if this has been fixed upstream or if an upstream bug has been filed (or if it would be useful to do so).

The sad conclusion is that for the next several years, if you need the local hostname without any dots on it you should write something like:

hostname | sed 's/\..*//'

instead of using the shorter, nicer hostname -s.

(We found this out the hard way last night, when we had some sort of network issue that made our DNS servers unreachable to some machines while some of our status check scripts were running.)