linux/ThreadMemoryUse written at 02:54:07; Add Comment
Why Linux threads use up so much memory
If you write a simple threaded program in C on a modern Linux system and fire it up, you may rapidly notice that you can't have more than a couple hundred threads before you run out of memory. Even if you don't need that many threads, you may discover that your program is using an absurd amount of (virtual) memory.
(This can also happen in other languages; I know it happens in Python.)
What's eating all the memory is the default NPTL thread stack size.
If your program doesn't specify a stack size for its new threads, NPTL
defaults to your 'stack size' resource limit (what '
(NPTL is the 'native Posix thread library' for Linux, designed to use the new kernel support routines that appeared in the 2.6 kernel series. If you're using a distribution based on the 2.6 kernel (or Red Hat 8 or later, including RHEL 3 I believe), you're almost certainly using NPTL for threads.)
There's two obvious ways out:
If you spawn other programs, beware of setting your stack size too low; 128K is about the minimum I would go these days. (I definitely saw core dumps at 64K.)
The gory hack variant of #2 is to crank the stack size resource limit
down before starting your program and then have your program crank it
right back up to where it should be once it gets control. This is
useful for environments (such as Python) where you don't have control
(In Python, you want the resource module.)
* * *
Atom feeds are available; see the bottom of most pages.