Why Go's big virtual size for 64-bit programs makes sense

December 22, 2014

In reaction to my entry on why your 64-bit Go programs are going to have a huge virtual size, sgoody on Hacker News asked why Go does this. There are two answers, depending on what 'this' you're talking about.

The reason that Go allocates all of this virtual memory address space is that it keeps other code from accidentally occupying some of it. This might be C runtime libraries that Go code winds up using or it might be Go code (yours or from packages) that calls mmap() directly or indirectly, and someday it will also be because of dynamically loaded Go code. If the Go runtime didn't explicitly fence off the address range it wanted to use, it could at least have the amount of memory it can allocate reduced by other people camping on bits of it. This is essentially standard practice; if you're going to want some chunk of address space later, you might as well fence it off now.

The reason to use this scheme for low level memory allocation is likely because it's simple, and simple is generally fast for memory allocators. Being fast is good here not just for the obvious reason, but also because this is a central low-level allocator and Go is a concurrent environment. You're probably going to have to take locks to use a central allocator, so the shorter the locks are held for the better. A fast central allocator is one that's unlikely to become a bottleneck and point of locking contention.

There are a number of reasons for Go not to just call the C runtime's malloc() to get memory (either at a low level or at a high one). Malloc is a very general allocator and as a result it may do all sorts of things that you don't need and don't want. It's also probably going to perform worse than a tuned custom allocator; in fact having your own allocator is extremely common in language runtimes, even for things like Python. Using it also means that you depend on the C runtime library, which is a problem when cross-compiling Go programs.

(Current versions of Go are also trying to do as much in Go code as possible instead of in C, partly because it makes the life of the garbage collector simpler.)

Written on 22 December 2014.
« A steady change in the source of blog comment spam attempts
The future of OmniOS here if we can't get 10G-T working on it »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Dec 22 00:38:35 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.