Go programs and Linux glibc versioning
Suppose that you build a Go program on one Linux machine, copy it to a second one, run it on that second machine, and get an error message like this:
./dlv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./dlv) ./dlv: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./dlv)
You might be puzzled. The direct explanation of what's going on is that you've built this Go program on a machine with a newer version of the GNU C library (glibc) than the second machine (for example, you built this program on a fast Ubuntu 22.04 compute server and tried to run it on an old Ubuntu 18.04 server, as I did here). For programs affected by this, the general rule of thumb is to build them on the oldest Linux distribution version that you expect to run them on.
At this point you might be confused; after all, Go programs are famous for being self-contained and statically linked. Well, usually they are but there are a surprising number of things that can make Go programs be dynamically linked. However this problem doesn't happen to all dynamically linked programs, just some of them. One answer for why is that it depends on what C library functions you use. Delve happens to indirectly use pthreads functions (somehow, I'm not sure exactly how), and these are the functions triggering this glibc versioning issue.
The more interesting and complex answer is that the GNU C Library
has a complex system of symbol versioning for backward compatibility,
known as 'compat symbols'. That's why there's two glibc versions
being mentioned here instead of just one; Delve uses pthreads things
that are at two different version levels in the system it was built
on (Ubuntu 22.04 in this case). We can use the '
line from that article to see what symbols Delve is depending on
at each version level:
; readelf --dyn-syms -W dlv | fgrep GLIBC_2.32 27: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pthread_sigmask@GLIBC_2.32 (7) ; readelf --dyn-syms -W dlv | fgrep GLIBC_2.34 3: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.34 (4) 26: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pthread_detach@GLIBC_2.34 (4) 34: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pthread_create@GLIBC_2.34 (4) 50: 0000000000000000 0 FUNC GLOBAL DEFAULT UND pthread_attr_getstacksize@GLIBC_2.34 (4)
As we see here, even the libc start function is versioned. However and fortunately, you don't necessarily get the most recent version of it when you link a program. For example, another Go program compiled on the same Ubuntu 22.04 machine as dlv has:
; readelf --dyn-syms -W gocritic | fgrep __libc_start_main 43: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.2.5 (3)
This Go binary will run fine on our Ubuntu 18.04 machines, unlike the Delve binary. And of course a lot of Go binaries will be statically linked.
If you want to increase the number of your Go binaries that are
statically linked, there are two options. The large scale and
potentially damaging option is to build with '
The problem with this is that some programs may not build or work
well, or they may be missing features. In Delve's case, I believe
that cgo is required for dlv trace's experimental ebpf support.
(Just turning off cgo apparently won't guarantee static binaries. If you want to build static binaries no matter what, see this 2020 comment and Go issue #26492.)
The more limited and flexible option is to eliminate the two most
common causes of dynamic linking. This is
done by using '
go install -tags osusergo,netgo ...', as covered
in places like Martin Tournoij's Statically compiling Go programs. As far as I know there's no
central Go reference for these build tags to avoid dynamic linking
in various situations; instead, you need to read the os/user and net package
documentation. Building this way will make a lot of programs into
statically linked ones while still letting programs that really want to
be dynamically linked for some reason do so.
(As of Go 1.18 I don't believe there are any other such 'use pure Go' build tags for standard packages.)
PS: Some of Delve's calls to versioned pthreads functions appear to come from runtime/cgo. See for example gcc_libinit.c and gcc_linux_amd64.c. I'm not sure why Delve is the only program affected by this out of everything I build.