Wandering Thoughts archives

Some things that force Go to call the C library for name resolution on Linux

Traditionally on Unix systems there is no official standard for how to do various sorts of network name lookups, or rather the official standard is 'call the functions in the system C library'. There is generally no actual specification for how name lookup works at the level that would permit you to create an independent implementation (although there is generally documentation on how to configure it). This presents a problem for people who are creating non-C languages; they must either arrange to call the C library (through the C library's limited interfaces for this) or write their own versions that may not resolve things exactly like the C library does, so that you get inconsistent behavior between C programs and programs written in the other language.

Go kind of takes both approaches. As covered in the net package's documentation, Go can both call the C library routines (using cgo) and do its own name lookups in pure Go. Go normally tries to use the pure Go approach as much as possible because it's considered better (partly because cgo calls can be relatively expensive). In theory the pure Go approach should give you the same results as the cgo approach; in practice, the two can behave somewhat differently in some situations, sometimes because of oversights.

(Although the net package's documentation talks only about DNS related lookups, this also affects how at least net.LookupPort() works.)

Go normally attempts to be pretty hyper-intelligent about whether or not it can use its pure Go lookup functions. It makes this decision in part by reading through your /etc/resolv.conf and /etc/nsswitch.conf to see if you're using anything that it doesn't think it can handle. This raises the question of what things in either of these files can accidentally force Go to use cgo calls to the C library, instead of its own more efficient (and more consistent across systems) pure Go version. For /etc/resolv.conf, Go understands all of the common things but anything else will force it to cgo, including any mistakes you may have lurking in there. For /etc/nsswitch.conf, Go looks at the 'hosts' line and a few complications can be common on modern Linuxes:

• if your hosts includes myhostname, only lookups of names with dots in them can be done in pure Go. Because of an implementation quirk, this currently means that net.LookupPort() is forced to use the C library.

  (Some other things are also forced to use the C library, but arguably they should in this situation because they involve hostnames.)

• if your hosts includes mymachines, all lookups go to the C library. This is probably common on modern systemd-based Linux distributions.

If you're using Go programs and you don't use containers or don't need the magic functionality of mymachines, you may want to strip it out of your nsswitch.conf. If you're like me, you may even be surprised to find it there in the first place. You may not want myhostname either, especially if your host has IP aliases that are most definitely not included in what a name to IP lookup for its hostname should return.

Note that contrary to what you might think, net.LookupPort() (and things that call it to get ports, like net.ResolveTCPAddr()) does not look at the services line in /etc/nsswitch.conf, only the hosts line. And of course the pure Go port lookup only looks at /etc/services (and may not parse it exactly like the C library does). At the moment a missing or unreadable /etc/services seems not to force a fallback to the C library but instead uses a tiny built in default version.

(This probably doesn't affect anyone. I can't imagine that there are very many people who use NIS or otherwise do something funny for services lookups and not hosts lookups.)

You can see what sort of lookup is used for specific queries by setting the GODEBUG environment variable to a verbosity of 2 or more, ie 'GODEBUG=netdns=2'. The resulting report may look something like this:

go package net: dynamic selection of DNS resolver
go package net: hostLookupOrder() = cgo
go package net: hostLookupOrder(smtp.cs) = files,dns

This is on a Linux machine where I set hosts to include myhostname but not mymachines. The first hostLookupOrder() is for looking up the port number for smtp; here the presence of myhostname forced it to resort to cgo. A blank argument to hostLookupOrder() is used by several sorts of lookups, including net.LookupAddr() and net.LookupCNAME().

(This is of course an implementation detail and may change at some point.)

Getting xterm and modern X applications to do cut and paste together

I was all set to write a somewhat grumpy blog entry about getting xterm and Chrome to play nice with each other for cut and paste when I decided to re-test this systematically. Let me tell you, systematic testing and investigation changes quite a lot of blog entries around here, and this one appears to be no exception.

The basic background on this is that X basically has two levels of cut and paste, which I'll call a casual level and a serious level. The casual level is making ordinary selections and pasting them with the middle mouse button (in most X applications), while the serious level is done through explicit Copy and Paste actions, generally found in menu entries. In X jargon, the 'casual level' is the PRIMARY selection and the 'serious level' is the CLIPBOARD selection.

(The history behind why both of these exist is long and tangled and basically goes back to the history of cut and paste in X and xterm's different cut and paste model.)

Xterm works fine at the ordinary selection level. Selections you make in xterm can be pasted into modern X applications like Firefox and Chrome, and selections that you sweep out in Chrome or Firefox can be pasted into xterm. But if you look at xterm, you will search in relative vain for a Copy or a Paste menu item. Modern terminal programs like gnome-terminal have entries right there and they do what you expect, but xterm does not.

There are two ways to get Copy in xterm. The first is the confusingly named 'Select to Clipboard' entry on the xterm menu you get with Control-middle mouse button; this makes all of your ordinary xterm selections be Copy-level selections instead of ordinary selections (ie, CLIPBOARD instead of PRIMARY, hence the name of the menu item). You can no longer paste them into most other things with the middle mouse button, but you can Paste them. The second is to add a keyboard binding for Copy in your X resources:

XTerm*VT100.Translations: #override \
     Ctrl Shift <KeyPress> C: copy-selection(CLIPBOARD)

The advantage of a keyboard binding is that regular selection stuff continues to work as it normally does; you just explicitly invoke the Copy operation whenever you need it. Among other things, this means you don't have to remember the 'Select to ...' mode that any particular xterm is in (or forget and be puzzled about why you may not be able to middle mouse button paste a selection in one xterm into something else).

Getting Paste in xterm is much more confusing, so I will give you the conclusion; if you want a real Paste, you need to add another keyboard override:

XTerm*VT100.Translations: #override \
     Ctrl Shift <KeyPress> C: copy-selection(CLIPBOARD) \n\
     Ctrl Shift <KeyPress> V: insert-selection(CLIPBOARD)

Xterm tries very hard to do something smart when you use the middle mouse button to paste things. As part of this smartness, it normally pastes the basic selection from whatever program has one; however, if there is no basic selection at the moment and there is a Copy-level selection, it will paste that instead. The time when you need an explicit Paste operation (as provided here with Shift-Control-V) is when the basic selection is different from what got Copy'd and you specifically want to paste the Copy'd stuff.

(In the jargon, xterm normally tries to paste the PRIMARY selection but if that doesn't exist, it's willing to try the CLIPBOARD. Our new key binding does an explicit insert from CLIPBOARD.)