Wandering Thoughts archives

All configuration files should support some form of file inclusion

Over on the Fediverse, I said something:

Every configuration file format should have a general 'include this file' feature, and it should support wildcards (for 'include subdir/*.conf'). Sooner or later people are going to need it, especially if your software gets popular.

It's unfortunate that standard YAML does not support this, although it's also sort of inevitable (YAML doesn't require files at all). This leaves everyone using YAML for their configuration file format to come up with various hacks.

(If this feature is hard-coded, it should use file extensions.)

There are a variety of reasons why people wind up wanting to split up a configuration file into multiple pieces. Obvious ones include that it's easier to coordinate multiple people or things wanting to add settings, a single giant file can be hard to read and deal with, and it's easy to write some parts by hand and automatically generate others. A somewhat less obvious reason is that this makes it easy to disable or re-enable an entire cluster of configuration settings; you can do it by simply renaming or moving around a file, instead of having to comment out a whole block in a giant file and then comment it back in later.

(All of these broadly have to do with operating the software in the large, possibly at scale, possibly packaged by one group of people and used by another. I think this is part of why file inclusion is often not an initial feature in configuration file formats.)

One of the great things about modern (Linux) systems and some modern software is the pervasive use of such 'drop-in' included configuration files (or sub-files, or whatever you want to call them). Pretty much everyone loves them and they've turned out to be very useful for eliminating whole classes of practical problems. Implementing them is not without issues, since you wind up having to decide what to do about clashing configuration directives (usually 'the last one read wins', and then you definite it that files are read in name-sorted order) and often you have to implement some sort of section merging (so that parts of some section can be specified in more than one file). But the benefits are worth it.

As mentioned, one subtle drawback of YAML as a configuration file format is that there's no general, direct YAML feature for 'include a file'. Programs that use YAML have to implement this themselves, by defining schemas that have elements with special file inclusion semantics, such as Prometheus's scrape_config_files: section in its configuration file, which lets you include files of scrape_config directives:

# Scrape config files specifies a list of globs.
# Scrape configs are read from all matching files
# and appended to the list of scrape configs.
scrape_config_files:
  [ - <filepath_glob> ... ]

That this only includes scrape_config directives and not anything else shows some of the limitations of this approach. And since it's not a general YAML feature, general YAML linters and so on won't know to look at these included files.

However, this sort of inclusion is still much better than not having any sort of inclusion at all. Every YAML based configuration file format should support something like it, at least for any configuration section that get large (for example, because it can have lots of repeated elements).