Why Go's big virtual size for 64-bit programs makes sense
In reaction to my entry on why your 64-bit Go programs are going to have a huge virtual size, sgoody on Hacker News asked why Go does this. There are two answers, depending on what 'this' you're talking about.
The reason that Go allocates all of this virtual memory address space
is that it keeps other code from accidentally occupying some of
it. This might be C runtime libraries that Go code winds up using or it might be Go code (yours or from packages)
mmap() directly or indirectly, and someday it will also
be because of dynamically loaded Go code. If the Go runtime didn't
explicitly fence off the address range it wanted to use, it could at
least have the amount of memory it can allocate reduced by other people
camping on bits of it. This is essentially standard practice; if you're
going to want some chunk of address space later, you might as well fence
it off now.
The reason to use this scheme for low level memory allocation is likely because it's simple, and simple is generally fast for memory allocators. Being fast is good here not just for the obvious reason, but also because this is a central low-level allocator and Go is a concurrent environment. You're probably going to have to take locks to use a central allocator, so the shorter the locks are held for the better. A fast central allocator is one that's unlikely to become a bottleneck and point of locking contention.
There are a number of reasons for Go not to just call the C runtime's
malloc() to get memory (either at a low level or at a high one).
Malloc is a very general allocator and as a result it may do all sorts
of things that you don't need and don't want. It's also probably
going to perform worse than a tuned custom allocator; in fact having
your own allocator is extremely common in language runtimes, even
for things like Python. Using it also means that you depend on the C
runtime library, which is a problem when cross-compiling Go programs.
(Current versions of Go are also trying to do as much in Go code as possible instead of in C, partly because it makes the life of the garbage collector simpler.)