Some things that force Go to call the C library for name resolution on Linux

December 14, 2015

Traditionally on Unix systems there is no official standard for how to do various sorts of network name lookups, or rather the official standard is 'call the functions in the system C library'. There is generally no actual specification for how name lookup works at the level that would permit you to create an independent implementation (although there is generally documentation on how to configure it). This presents a problem for people who are creating non-C languages; they must either arrange to call the C library (through the C library's limited interfaces for this) or write their own versions that may not resolve things exactly like the C library does, so that you get inconsistent behavior between C programs and programs written in the other language.

Go kind of takes both approaches. As covered in the net package's documentation, Go can both call the C library routines (using cgo) and do its own name lookups in pure Go. Go normally tries to use the pure Go approach as much as possible because it's considered better (partly because cgo calls can be relatively expensive). In theory the pure Go approach should give you the same results as the cgo approach; in practice, the two can behave somewhat differently in some situations, sometimes because of oversights.

(Although the net package's documentation talks only about DNS related lookups, this also affects how at least net.LookupPort() works.)

Go normally attempts to be pretty hyper-intelligent about whether or not it can use its pure Go lookup functions. It makes this decision in part by reading through your /etc/resolv.conf and /etc/nsswitch.conf to see if you're using anything that it doesn't think it can handle. This raises the question of what things in either of these files can accidentally force Go to use cgo calls to the C library, instead of its own more efficient (and more consistent across systems) pure Go version. For /etc/resolv.conf, Go understands all of the common things but anything else will force it to cgo, including any mistakes you may have lurking in there. For /etc/nsswitch.conf, Go looks at the 'hosts' line and a few complications can be common on modern Linuxes:

  • if your hosts includes myhostname, only lookups of names with dots in them can be done in pure Go. Because of an implementation quirk, this currently means that net.LookupPort() is forced to use the C library.

    (Some other things are also forced to use the C library, but arguably they should in this situation because they involve hostnames.)

  • if your hosts includes mymachines, all lookups go to the C library. This is probably common on modern systemd-based Linux distributions.

If you're using Go programs and you don't use containers or don't need the magic functionality of mymachines, you may want to strip it out of your nsswitch.conf. If you're like me, you may even be surprised to find it there in the first place. You may not want myhostname either, especially if your host has IP aliases that are most definitely not included in what a name to IP lookup for its hostname should return.

Note that contrary to what you might think, net.LookupPort() (and things that call it to get ports, like net.ResolveTCPAddr()) does not look at the services line in /etc/nsswitch.conf, only the hosts line. And of course the pure Go port lookup only looks at /etc/services (and may not parse it exactly like the C library does). At the moment a missing or unreadable /etc/services seems not to force a fallback to the C library but instead uses a tiny built in default version.

(This probably doesn't affect anyone. I can't imagine that there are very many people who use NIS or otherwise do something funny for services lookups and not hosts lookups.)

You can see what sort of lookup is used for specific queries by setting the GODEBUG environment variable to a verbosity of 2 or more, ie 'GODEBUG=netdns=2'. The resulting report may look something like this:

go package net: dynamic selection of DNS resolver
go package net: hostLookupOrder() = cgo
go package net: hostLookupOrder(smtp.cs) = files,dns

This is on a Linux machine where I set hosts to include myhostname but not mymachines. The first hostLookupOrder() is for looking up the port number for smtp; here the presence of myhostname forced it to resort to cgo. A blank argument to hostLookupOrder() is used by several sorts of lookups, including net.LookupAddr() and net.LookupCNAME().

(This is of course an implementation detail and may change at some point.)


Comments on this page:

By Jonathan at 2015-12-17 09:28:28:

I suspect this is the reason that I find docker (written in Go) caches NXDOMAIN and doesn't seem to let go of it. I hit this as I pull/push to a docker registry behind a VPN, and if the daemon tries to resolve my registry host when the VPN is down, I have to restart the daemon after restarting the VPN in order for it to resolve the host correctly.

Written on 14 December 2015.
« Getting xterm and modern X applications to do cut and paste together
I just had another smooth Fedora version upgrade with ZFS on Linux »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Dec 14 22:39:43 2015
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.