Wandering Thoughts archives

Why Linux threads use up so much memory

If you write a simple threaded program in C on a modern Linux system and fire it up, you may rapidly notice that you can't have more than a couple hundred threads before you run out of memory. Even if you don't need that many threads, you may discover that your program is using an absurd amount of (virtual) memory.

(This can also happen in other languages; I know it happens in Python.)

What's eating all the memory is the default NPTL thread stack size. If your program doesn't specify a stack size for its new threads, NPTL defaults to your 'stack size' resource limit (what 'ulimit -s' reports in bash, aka RLIMIT_STACK); if your stack size is set to unlimited, NPTL picks two megabytes. NPTL only looks at your RLIMIT_STACK once, very early during program startup (well before the C main() gets control).

(NPTL is the 'native Posix thread library' for Linux, designed to use the new kernel support routines that appeared in the 2.6 kernel series. If you're using a distribution based on the 2.6 kernel (or Red Hat 8 or later, including RHEL 3 I believe), you're almost certainly using NPTL for threads.)

There's two obvious ways out:

1. set a non-default stack size via pthread_attr_init and pthread_attr_setstack, if you have access to that level of control over pthread_create's usage.
2. crank your stack size resource limit down before starting the program; this works on anything.

If you spawn other programs, beware of setting your stack size too low; 128K is about the minimum I would go these days. (I definitely saw core dumps at 64K.)

The gory hack variant of #2 is to crank the stack size resource limit down before starting your program and then have your program crank it right back up to where it should be once it gets control. This is useful for environments (such as Python) where you don't have control over pthread_create parameters but you can fiddle with the resource limits.

(In Python, you want the resource module.)

Modifying Linux kernel configuration options the easy way

Modifying a Linux kernel configuration option can be an annoying experience, as you poke around the menus of 'make menuconfig' trying to find where the configuration option you want to change is hiding. Fortunately there's a less well known but much easier way.

The kernel build process stores configuration settings in the file .config in the root of the kernel source tree. However, it generates other things from the file (and some options are interdependent), so you can't just edit the file and go; you need to resynchronize the build system with the .config.

So the easy way to modify an existing configuration is the following:

1. Find out what the CONFIG_ variable you want to change is. Many of these have obvious names, like CONFIG_SMP.
2. Edit .config to delete any mention of the option, even if this is in a comment. (Comments are significant in .config files.)
3. Run 'make oldconfig', which will stop to ask you about the missing options at appropriate moments.
4. Build.

Finding the right CONFIG_ variable is a bit tricky. I will just say that they are all specified in Kconfig files in various source directories (for example, net/Kconfig), which is also where you find all of the help text. (A few Kconfig files are called things like fs/Kconfig.binfmt.)

'make oldconfig' is also what you use if you're migrating a .config from one Linux kernel version to another, since it will prompt you about any new configuration options (including new drivers). In fact, any time you change or shuffle .configs around, you want to do 'make oldconfig'; it is the easiest way of getting the kernel build system synchronized with the .config.

PS: remember to save your .config file before you do a 'make mrproper' to clean out kernel source trees; it's one of the files that a full cleaning deletes.

PPS: many distributions ship kernel source with their .config files stored somewhere; for example, Red Hat (including Fedora Core) puts them in configs/*. They make good starting .configs, saving you the tedium of 'make menuconfig' from scratch.