I've mentioned that we ran out of inodes on a Linux server and covered what the high level problem was, but I've never described the actual mechanics of how and why you can run out of inodes on a filesystem, or more specifically on an extN filesystem. I have to be specific about the filesystem type, because how this is handled varies from filesystem to filesystem; some either have no limit on how many inodes you can have or have such a high limit that you're extremely unlikely to run into it.

The fundamental reason you can run out of inodes on an extN filesystem is that extN statically allocates space for inodes; in every extN filesystem, there is space for so many inodes reserved, and you can never have any more than this. If you use 'df -i' on an extN filesystem, you can see this number for the filesystem, and you can also see it with dumpe2fs, which will tell you other important information. Here, let's look at an ext4 filesystem:

# dumpe2fs -h /dev/md10
[...]
Block size:               4096
[...]
Blocks per group:         32768
[...]
Inodes per group:         8192
[...]

I'm showing this information because it leads to the important parameter for how many inodes any particular extN filesystem has, which is the bytes/inode ratio (mke2fs's -i argument). By default this is 16 KB, ie there will be one inode for every 16 KB of space in the filesystem, and as the mke2fs manpage covers, it's not too sensible to set it below 4 KB (the usual extN block size).

The existence of the bytes/inode ratio gives us a straightforward answer for how you can run a filesystem out of inodes: you simply create lots of files that are smaller than this ratio. ExtN implicitly assumes that each inode will on average use at least 16 KB of disk space; if on average your inodes use less, you will run out of inodes before you run out of disk space. One tricky thing here is that this space doesn't have to be used up by regular files, because other sorts of inodes can be small too. Probably the easiest other source is directories; if you have lots of directories with a relatively small number of subdirectories and files in each, it's quite possible for many of them to be smaller than 16 KB, and in some cases you can have a great many subdirectories.

(In our problem directory hierarchy, almost all of the directories are 4 KB, although a few are significantly larger. And the hierarchy can have a lot of subdirectories when things go wrong.)

Another case is symbolic links. Most symbolic links are quite small, and in fact ext4 may be able to store your symbolic link entirely in the inode itself. This means that you can potentially use up a lot of inodes without using any disk space (well, beyond the space for the directories that the symbolic links are in). There are other sorts of special files that also use little or no disk space, but you probably don't have tons of them in an extN filesystem unless something unusual is going on.

(If you do have tens of thousands of Unix sockets or FIFOs or device files, though, you might want to watch out. Or even tons of zero-length regular files that you're using as flags and a persistence mechanism.)

Most people will never run into this on most filesystems, because most filesystems have an average inode size usage that's well above 16 KB. There usually plenty of files over 16 Kb, not that many symbolic links, and a relatively few (small) directories compared to the regular files. For instance, one of my relatively ordinary Fedora root filesystem has a bytes/inode ratio of roughly 73 Kb per inode, and another is at 41 KB per inode.

(You can work out your filesystem's bytes/inode ratio simply by dividing the space used in KB by the number of inodes used.)