Wandering Thoughts archives

Why Go cares about the difference between unsafe.Pointer and uintptr

Go has two things that are more or less the representation of an untyped pointer; uintptr and unsafe.Pointer (which, contrary to appearances, is a built-in type). On the surface this is a little bit odd, because unsafe.Pointer and uintptr can be converted back and forth between each other. Why not have only one representation of a pointer? What's the difference between the two?

The superficial difference is that you can do arithmetic on an uintptr but not on an unsafe.Pointer (or any other Go pointer). The important difference is explicitly pointed out by the unsafe package's documentation:

A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.

Although unsafe.Pointers are generic pointers, the Go garbage collector knows that they point to Go objects; in other words, they are real Go pointers. Through internal magic, the garbage collector can and will use them to keep live objects from being reclaimed and to discover further live objects (if the unsafe.Pointer points to an object that has pointers of its own). Due to this, a lot of the restrictions on what you can legally do with unsafe.Pointers boil down to 'at all times, they must point to real Go objects'. If you create an unsafe.Pointer that doesn't, even for a brief period of time, the Go garbage collector may choose that moment to look at it and then crash because it found an invalid Go pointer.

By contrast, a uintptr is merely a number. None of this special garbage collection magic applies to objects 'referred to' by a uintptr because as just a number, a uintptr doesn't refer to anything. In turn this leads to many of the careful limitations on the various ways that you can turn an unsafe.Pointer into a uintptr, manipulate it, and turn it back. The basic requirement is that you do this manipulation in such a way that the compiler and the runtime can shield the temporary non-pointerness of your unsafe.Pointer from the garbage collector so this temporary conversion will be atomic with respect to garbage collection.

(I think that my use of unsafe.Pointer in my entry on copying blobs of memory into Go structures is safe, but I admit I'm now not completely sure. I believe that there is some magic going on with cgo, since it can safely manufacture unsafe.Pointers that point to C memory, not Go memory.)

PS: As of Go 1.8, all Go pointers must always be valid (I believe including unsafe.Pointers), even if a garbage collection isn't running at the time. If you ever have an invalid pointer stored in a variable or a field, your code can crash merely by updating the field to a perfectly valid value, including nil. See for example this educational Go bug report.

(I was going to try to say something about the internal magic that allows the garbage collector to cope with untyped unsafe.Pointer pointers, but I'm convinced I don't understand enough about it to even say what sort of magic it uses.)

A recent performance surprise with X on my Fedora Linux desktop

As I discussed yesterday, on Monday I 'upgraded' my office workstation by transplanted my system disks and data disks from my old office hardware to my new office hardware. When I turned the new hardware on, my Fedora install booted right up pretty much exactly as it had been (some advance planning made networking work out), I logged in on the text console, started up X, and didn't think twice about it. Modern X is basically configuration free and anyway, both the old and the new hardware had Radeon cards (and the same connectors for my monitors, so my dual-screen setup wouldn't get scrambled). I even ran various OpenGL test programs to exercise the new card and see if it would die under far more demanding load than I expected to ever put on it.

(This wound up leading to some lockups.)

All of this sounds perfectly ordinary, but actually I left out an important detail that I only discovered yesterday. My old graphics card is a Radeon HD 5450, which uses the X radeon driver. My new graphics card is a Radeon RX 550, but things have changed since 2011 so it uses the more modern amdgpu driver. And I didn't have the amdgpu driver installed in my Fedora setup (like most X drivers, it's in a separate RPM of its own), so the X server was using neither the amdgpu driver (which it didn't have) nor the radeon driver (which doesn't support the RX 550).

The first surprise is that X worked anyways and I didn't notice anything particular wrong or off about my X session. Everything worked and was as responsive as I expected, and the OpenGL tests I ran seemed to go acceptably fast (as did a full-screen video). In retrospect there were a few oddities that I noticed as I was trying things due to my system hangs (xdriinfo reported no direct rendering and vdpauinfo spat out odd errors, for example), but there was nothing obvious (and glxinfo reported plausible things).

The second surprise is what X was actually using to drive the display, which turns out to be something called the modesetting driver. This driver is a quite basic one that relies on kernel mode setting but is otherwise more or less unaccelerated. Well, sort of, because modesetting was apparently using glamor to outsource some rendering to OpenGL, in case you have hardware accelerated OpenGL, which I think that I did in this setup. I'm left unsure of how much hardware acceleration I was getting; maybe my CPU was rendering 24-bit colour across two 1920x1200 LCDs without me noticing, or maybe a bunch of it was actually hardware accelerated even with a generic X driver.

(There is a tangled web of packages here. I believe that the open source AMD OpenGL code is part of the general Mesa packages, so it's always installed if you have Mesa present. But I don't know if the Mesa code requires the X server to have an accelerated driver, or if a kernel driver is good enough.)

PS: Kernel mode setting was available because the kernel also has an amdgpu driver module that's part of the DRM system. That module is in the general kernel-modules package, so it's installed on all machines and automatically loaded whenever the PCI IDs match.

PPS: Given that I had system lockups before I installed the X server amdgpu driver, the Fedora and freedesktop bugs are really a kernel bug in the admgpu kernel driver. Perhaps this is unsurprising and already known.