2020-09-15
When the Go garbage collector will panic over bad pointer values
For some time, I've vaguely remembered that the Go garbage collector
actually checked Go pointer values and would panic if it found that
an alleged pointer (including unsafe.Pointer values) didn't point to a
valid object. Since the garbage collector may interrupt you at
almost random points, this would make it very dangerous to play
around with improper unsafe.Pointer values. However, this was
just a superstitious memory, so today
I decided to find out what the situation is in current Go by reading
the relevant runtime source code (for the development version of Go,
which is just a bit more recent than Go 1.15 as I write this).
As described in Allocator Wrestling (see also, and), Go allocates ordinary things (including goroutine stacks) from chunks of memory called spans that are themselves allocated as part of arenas. Arenas (and spans) represent address space that is used as part of the Go heap, but they may not currently have all of their memory allocated from the operating system. A Go program always has at least one arena created as part of its address space.
Based on reading the code, I believe that the Go garbage collector
panics if it finds a Go pointer that points inside a created arena
but is not within the bounds of a span that is currently in use
(including spans used for stacks). The Go garbage collector completely
skips checking pointers that don't fall within a created arena; the
comment in the source code says '[t]his pointer may be to some
mmap'd region, so we allow it', which might lead you to think that
it's talking about your potential use of mmap(), but the Go runtime
itself allocates a number of things outside of arenas in things
it has mmap()'d and obviously the garbage collector can't panic
over pointers to them.
The address space available on 64-bit machines is very large and many Go programs will use only a small portion of it for created arenas. The practical consequence of this is that many random 'pointer' values will not fall within the bounds of your program's arenas and so won't trigger garbage collector panics. You're probably more likely to produce these panics if you start with valid Go pointers and then manipulate them in sufficiently improper ways (but not so improperly that the pointer value flies off too far).
(So my superstitious belief has some grounding in reality but was
probably way too broad. It's certainly not safe to put bad values
in unsafe.Pointers, but in practice most bad values won't be
helpfully diagnosed with panics from the garbage collector; instead
you'll get other, much more mysterious issues when you try to use
them for real.)
An additional issue is that spans are divided up into objects, not
all of which are necessarily allocated at a given time. The current
version of the garbage collector doesn't seem to attempt to verify
that all pointers point to allocated objects inside spans, so I
believe that if you're either lucky or very careful in your
unsafe.Pointer manipulation, you can create a non-panicing pointer
to a currently free object that will later be allocated and used
by someone else.
(It's possible that such a pointer could cause garbage collector panics later on under some circumstances.)
The Go runtime also contains a much simpler pointer validity check (and panic) in the code that handles copying and adjusting goroutine stacks when they have to grow. This simply looks for alleged pointers that have a value that's 'too small' (but larger than 0), where too small is currently 4096. I believe that such bad pointers will pass the garbage collector's check, because they point well outside any created arena.
Both of these panics can be turned off with the same setting in
$GODEBUG, as covered in the documentation for the runtime
package. As you would expect,
the setting you want is 'invalidptr=0'.
People who want to see the code for this should look in
runtime/mbitmap.go's findObject(),
runtime/mheap.go's spanOf(),
and runtime/stack.go's adjustpointers().