Systemd's NSS myhostname module surprised me recently

March 31, 2021

The other day I did a plain traceroute from my Fedora 33 office workstation (instead of my usual 'traceroute -n') and happened to notice that the first hop was being reported as '_gateway'. This is very much not the name associated with that IP address, so I was rather surprised and annoyed. Although I initially suspected systemd-resolved because of a Fedora 33 change to use it, the actual cause turned out to be the myhostname NSS module, which was listed relatively early in the hosts: line in my nsswitch.conf.

(However, it turns out that I would probably have seen the same thing if I actually was using systemd-resolved, which I'm not.)

If configured in nsswitch.conf, the myhostname module provides three services, only two of which have to do with your machine's hostname. The simplest one is that localhost and variants on it all resolve to the appropriate localhost IPv4 and IPv6 addresses, and those localhost IPv4 and IPv6 addresses resolve back to 'localhost' in gethostbyaddr() and its friends. The second one is that the exact system host name resolves to all of the IP addresses on all of your interfaces; this is the name that hostname prints, and nothing else. Shortened or lengthened variants of the hostname don't do this. As with localhost, all of these IP addresses also resolve back to the exact local host name. This is where the first peculiarity comes up. To quote the documentation:

  • The local, configured hostname is resolved to all locally configured IP addresses ordered by their scope, or — if none are configured — the IPv4 address (which is on the local loopback) and the IPv6 address ::1 (which is the local host).

If you do a reverse lookup on, myhostname will always report that it has the name of your machine, even if you have configured IP addresses and so myhostname would not give you as an IP address for your hostname. A reverse lookup of ::1 will report that it's called both 'localhost' and your machine's name.

The third service is that the hostname "_gateway" is resolved to all current default routing gateway addresses. As with the first two services, the IP addresses of these gateways will also be resolved to the name "_gateway", which is what I stumbled over when I actually paid attention to the first hop in my traceroute output.

The current manpage for myhostname doesn't document that it also affects resolving IP addresses into names as well as names into IP addresses. A very charitable person could say that this is implied by saying that various hostnames 'are resolved' into IP addresses, as proper resolution of names to IP addresses implies resolving them the other way too.

Which of these effects trigger for you depends on where myhostname is in your nsswitch.conf. For instance, if it's present at all (even at the end), the special hostname "_gateway" will resolve to your gateway IPs, and names like "aname.localhost" will resolve to your IPv4 and IPv6 localhost IPs (and probably will resolve to your hostname). If it's present early, it will steal the resolution of more names and more IPs from DNS and other sources.

The myhostname NSS module is part of systemd and has worked like this for over half a decade (although it started out using "gateway" instead of "_gateway"). However, it's not necessarily packaged, installed, and configured along with the rest of systemd. Ubuntu splits it out into a separate package, libnss-myhostname, which isn't installed on our Ubuntu servers. Fedora packages it as part of 'systemd-libs', which means it's guaranteed to be installed, and appears to default to using it in nsswitch.conf.

(What I believe is a stock installed Fedora 33 VM image has a nsswitch.conf hosts: line of "files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns". You might think that this would make DNS results from systemd-resolved take precedence over myhostname, but in a quiet surprise systemd-resolved does this too; see the "Synthetic Records" section in systemd-resolved.service.)

PS: I don't know why I never noticed this special _gateway behavior before, since myhostname has been doing all of this for some time (and I've had it in my nsswitch.conf ever since Fedora started shoving it in there). Possibly I just never noticed the name of the first hop when I ran plain 'traceroute', because I always knew what it was.

PPS: The change from "gateway" to "_gateway" happened in systemd 235, released 2017-10-06. The "gateway" feature for myhostname was introduced in systemd 218, released 2014-12-10. All of this is from systemd's NEWS file.

Comments on this page:

From at 2021-03-31 06:56:03:

While we're talking about nss-myhostname, here's another unexpected thing about the code which handles _gateway:

I tinker with BGP routing and have a full IPv6 BGP feed from a few free providers. That's not nearly as large as the IPv4 one, but it adds up to about 200 MB of memory usage in my case. Since this is a hobby thing, I'm just doing it on Linux, with BIRD running alongside other regular services such as Postfix and httpd on the same machine.

Now when nss-myhostname needs to check whether a given IP address should resolve to _gateway, you would think it just queries the default route from the kernel. No. It queries all the routes. From all routing tables.

As a result, if your routing tables add up to 200 MB, then every process which called gethostbyaddr or getnameinfo will itself grow by 200 MB, all of it allocated by nss-myhostname to store the received routes. (Fortunately that's one-time and doesn't add up per call, but on tiny VPSes, 200 MB per process is still a lot.)

To be fair, this isn't entirely a problem with the systemd module itself, as for a long time the kernel didn't even offer any way to dump just a subset of routes via netlink, so tools such as ip route had to drink the whole thing and filter it client-side.

(Eventually, route filtering was implemented in the kernel, but for some weird reason it only works for ip route but I couldn't get it to work for nss-myhostname despite them making seemingly identical netlink queries.)

So anyway, don't use nss-myhostname on routers, at least for the time being, or it will cause mysterious RAM leaks.

Written on 31 March 2021.
« Nil in Go is typed in theory and sort of untyped in practice
Understanding Prometheus' changes() function and what it can do for me »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 31 00:46:16 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.