Go programs and Linux glibc versioning

June 14, 2022

Suppose that you build a Go program on one Linux machine, copy it to a second one, run it on that second machine, and get an error message like this:

./dlv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./dlv)
./dlv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./dlv)

You might be puzzled. The direct explanation of what's going on is that you've built this Go program on a machine with a newer version of the GNU C library (glibc) than the second machine (for example, you built this program on a fast Ubuntu 22.04 compute server and tried to run it on an old Ubuntu 18.04 server, as I did here). For programs affected by this, the general rule of thumb is to build them on the oldest Linux distribution version that you expect to run them on.

At this point you might be confused; after all, Go programs are famous for being self-contained and statically linked. Well, usually they are but there are a surprising number of things that can make Go programs be dynamically linked. However this problem doesn't happen to all dynamically linked programs, just some of them. One answer for why is that it depends on what C library functions you use. Delve happens to indirectly use pthreads functions (somehow, I'm not sure exactly how), and these are the functions triggering this glibc versioning issue.

The more interesting and complex answer is that the GNU C Library has a complex system of symbol versioning for backward compatibility, known as 'compat symbols'. That's why there's two glibc versions being mentioned here instead of just one; Delve uses pthreads things that are at two different version levels in the system it was built on (Ubuntu 22.04 in this case). We can use the 'readelf' command line from that article to see what symbols Delve is depending on at each version level:

; readelf --dyn-syms -W dlv | fgrep GLIBC_2.32
27: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask@GLIBC_2.32 (7)
; readelf --dyn-syms -W dlv | fgrep GLIBC_2.34
 3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34 (4)
26: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_detach@GLIBC_2.34 (4)
34: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_create@GLIBC_2.34 (4)
50: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_attr_getstacksize@GLIBC_2.34 (4)

As we see here, even the libc start function is versioned. However and fortunately, you don't necessarily get the most recent version of it when you link a program. For example, another Go program compiled on the same Ubuntu 22.04 machine as dlv has:

; readelf --dyn-syms -W gocritic | fgrep __libc_start_main
43: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (3)

This Go binary will run fine on our Ubuntu 18.04 machines, unlike the Delve binary. And of course a lot of Go binaries will be statically linked.

If you want to increase the number of your Go binaries that are statically linked, there are two options. The large scale and potentially damaging option is to build with 'CGO_ENABLED=0'. The problem with this is that some programs may not build or work well, or they may be missing features. In Delve's case, I believe that cgo is required for dlv trace's experimental ebpf support.

(Just turning off cgo apparently won't guarantee static binaries. If you want to build static binaries no matter what, see this 2020 comment and Go issue #26492.)

The more limited and flexible option is to eliminate the two most common causes of dynamic linking. This is done by using 'go install -tags osusergo,netgo ...', as covered in places like Martin Tournoij's Statically compiling Go programs. As far as I know there's no central Go reference for these build tags to avoid dynamic linking in various situations; instead, you need to read the os/user and net package documentation. Building this way will make a lot of programs into statically linked ones while still letting programs that really want to be dynamically linked for some reason do so.

(As of Go 1.18 I don't believe there are any other such 'use pure Go' build tags for standard packages.)

PS: Some of Delve's calls to versioned pthreads functions appear to come from runtime/cgo. See for example gcc_libinit.c and gcc_linux_amd64.c. I'm not sure why Delve is the only program affected by this out of everything I build.

Written on 14 June 2022.
« In general Unix system calls are not cancellable, just abortable
Understanding some peculiarities of per-cgroup memory usage accounting »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Jun 14 22:36:29 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.