Why Linux threads use up so much memory
If you write a simple threaded program in C on a modern Linux system and fire it up, you may rapidly notice that you can't have more than a couple hundred threads before you run out of memory. Even if you don't need that many threads, you may discover that your program is using an absurd amount of (virtual) memory.
(This can also happen in other languages; I know it happens in Python.)
What's eating all the memory is the default NPTL thread stack size.
If your program doesn't specify a stack size for its new threads, NPTL
defaults to your 'stack size' resource limit (what '
reports in bash, aka
RLIMIT_STACK); if your stack size is set to
unlimited, NPTL picks two megabytes. NPTL only looks at your
RLIMIT_STACK once, very early during program startup (well before
main() gets control).
(NPTL is the 'native Posix thread library' for Linux, designed to use the new kernel support routines that appeared in the 2.6 kernel series. If you're using a distribution based on the 2.6 kernel (or Red Hat 8 or later, including RHEL 3 I believe), you're almost certainly using NPTL for threads.)
There's two obvious ways out:
- set a non-default stack size via
pthread_attr_setstack, if you have access to that level of control over
- crank your stack size resource limit down before starting the program; this works on anything.
If you spawn other programs, beware of setting your stack size too low; 128K is about the minimum I would go these days. (I definitely saw core dumps at 64K.)
The gory hack variant of #2 is to crank the stack size resource limit
down before starting your program and then have your program crank it
right back up to where it should be once it gets control. This is
useful for environments (such as Python) where you don't have control
pthread_create parameters but you can fiddle with the
(In Python, you want the resource module.)