Linux Ext4 directories have a maximum size (in entries)

February 16, 2023

Suppose, not entirely hypothetically, that one day you discover your (Linux) kernel logging messages like this:

EXT4-fs warning (device md1): ext4_dx_add_entry:2461: Directory (ino: 102236164) index full, reach max htree level :2
EXT4-fs warning (device md1): ext4_dx_add_entry:2465: Large directory feature is not enabled on this filesystem

Congratulations, of a sort. You've managed to accumulate so many files in the directory that it has filled up, in a logical sense. Unfortunately, further attempts to create files will fail; in fact they are already failing, because that's how you get the error message. If you're lucky, your software is logging error messages and you're noticing them. Also, since your directory got so large, you may have an unpleasant surprise coming your way.

('Large' here is relative. The directory that this happened to was only about 575 MBytes as reported by 'ls -lh'; this is very large for a directory, but not that large for a modern file. The filesystem as a whole had tons of space free.)

You might reasonably ask how a directory can fill up when it's not near even the 2 GByte 32-bit file size limit and the filesystem has plenty of disk space and inodes left. What's going on is that in modern filesystems, (big) directories aren't just linear lists of entries; instead they're some sort of tree structure. In ext4 these are called hash tree ('htree') directories. When ext4 is adding an entry to such a htree, under some circumstances it can need to 'split (the) index' (according to code comments) by adding another level to the tree. However, ext4 has a maximum allowed number of levels that can be in the tree. If ext4 needs to add a level and can't because you're already at the maximum level, it reports the kernel error we're seeing here.

(The direct error message is in fs/ext4/namei.c, in ext4_dx_add_entry(), but to understand it you need to know something about ext4 htrees.)

I believe that this is a limit on the total number of entries you can have in a directory (instead of, say, a limit on the number of entries with some hash value or range of them). Some reading (cf) suggests that the normal limit is about 10 million files in a single directory if you don't have the ext4 'large directories' feature turned on.

(The documentation for large_dir in ext4(5) doesn't give specific numbers, and I believe it depends on your filesystem's block size as well, cf; our filesystem had 4K blocks, the default. Filesystems with large directories can have a three-level htree, but filesystems without the feature are limited to a two-level htree.)

If you're really running a system that should have that many files in a single directory, you need to turn on the 'large_dir' ext4 feature somehow (the tune2fs(8) manual page says this can be turned on without remaking the filesystem). Otherwise, you need to figure out what's gone wrong with either your system or your understanding of how it works, then change things so that it's not trying to put more than ten million files in one directory. Even if you can turn on large directories, you'll probably be happier fixing the underlying issue.

Written on 16 February 2023.
« BIOS MBR booting isn't always flawless and can be odd
Implementing 'grow down' window placement in Fvwm on X11 »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Feb 16 22:58:58 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.