Go basically never frees heap memory back to the operating system

October 6, 2018

Over on Reddit's r/golang, I ran into an interesting question about Go's memory use as part of this general memory question:

[...] However Go is not immediately freeing the memory, at least from htop's perspective.

What can I do to A) gain insight on when this memory will be made available to the OS, [...]

The usual question about memory usage in Go programs is when things will be garbage collected (which can be tricky). However, this person wants to know when Go will return free memory back to the operating system. This is a good question partly because programs often don't do very much of this (or really we should say the versions of malloc() that programs use don't do this), for various reasons. Somewhat to my surprise, it turns out that Go basically never returns memory address space to the OS, as of Go 1.11. In htop, you can expect normal Go programs to only ever be constant sized or grow, never to shrink.

(The qualification about Go 1.11 is important, because Go's memory handling changes over time. Back in 2014 with Go 1.5 or so, Go processes used a huge amount of virtual memory, but that's changed since then.)

The Go runtime itself initially allocates memory in relatively decent sized chunks of memory called 'spans', as discussed in the big comment at the start of runtime/malloc.go (and see also this and this (also)); spans are at least 8 KB, but may be larger. If a span has no objects allocated in it, it is an idle span; how many bytes are in idle spans is in runtime.MemStats.HeapIdle. If a span is idle for sufficiently long, the Go runtime 'releases' it back to the OS, although this doesn't mean what you think. Released spans are a subset of idle spans; when a span is released, it still counts as idle.

(In theory the number of bytes of idle spans released back to the operating system is runtime.MemStats.HeapReleased, but you probably want to read the comment about this in the source code of runtime/mstats.go.)

Counting released spans as idle sounds peculiar until you understand something important; Go doesn't actually give any memory address space back to the OS when a span is released. Instead, what Go does is to tell the OS that it doesn't need the contents of the span's memory pages any more and the OS can replace them with zero bytes at its whim. So 'released' here doesn't mean 'return the memory back to the OS', it means 'discard the contents of the memory'. The memory itself remains part of the process and counts as part of the process size (it may or may not count as part of the resident set size, depending on the OS), and Go can immediately use such a released idle span again if it wants to, just as it can a plain idle span.

(On Unix, releasing pages back to the OS consists of calling madvise() (Linux, FreeBSD) on them with either MADV_FREE or MADV_DONTNEED, depending on the specific Unix. On Windows, Go uses VirtualFree() with MEM_DECOMMIT. On versions of Linux with MADV_FREE, I'm not sure what happens to your RSS after doing it; some sources suggest that your RSS doesn't go down until the kernel starts actually reclaiming the pages from you, which may be some time later.)

As far as I can tell from inspecting the current runtime code, Go only very rarely returns memory that it has used back to the operating system by calling munmap() or the Windows equivalent. In particular, once Go has used memory for regular heap allocations it will never be returned to the OS even if Go has plenty of released idle memory that's been untouched for a very long time (as far as I can tell). As a result, the process virtual size that you see in tools like htop is basically a high water mark, and you can expect it to never go down. If you want to know how much memory your Go program is really using, you need to carefully look at the various bits and pieces in runtime.MemStats, perhaps exported through net/http/pprof.

Written on 06 October 2018.
« My non-approach to password management tools
A deep dive into the OS memory use of a simple Go program »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sat Oct 6 00:04:48 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.