i'd be interested in your thoughts on this post on rachelbythebay's blog:

So, okay, fine, at some point it made sense to run programs to empirically determine what was supported on a given system. What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over?

Why didn't we end up with a situation where it was just a standard thing that had a small number of possible values, and it would just be set for you somewhere? Whoever was responsible for building your system (OS company, distribution packagers, whatever) could leave something in /etc that says "X = flavor 1, Y = flavor 2" and so on down the line.

...

[I]t's still normal to ship a .tar.gz with an absolute crap-ton of dumb little macro files that run all kinds of inscrutable tests that give you the same answers that they did the last time they ran on your machine or any other machine like yours, and WILL give the same answers going forward.

That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable.

There has been some movement toward a style of almost statically known package configuration, with tools like pkg-config. Where there are API differences between versions of packages, these tools could presumably be extended to provide ways to report this (I think today some of this is done by versioning the package name).

However, broadly rachelbythebay's post is focused on what I could call the broad system API, where you can assume that most or all systems of this type (say 'Debian version X') have the same APIs with the same behavior, so there's no point in testing for it over and over. There is another important class of things that configure can test for, which covers optional APIs, like whether or not IPMI libraries are available, and behavior that isn't even exposed as an API (eg, the specific kernel version and its internal behavior). These are more or less always going to need some sort of specific dynamic check, even if the check is 'does pkg-config know about IPMI library X and can tell us how to use it'.

Configure's dynamic probing is also mechanism neutral (and cross system), which is an important consideration in practice because the open source world almost never agrees on a single mechanism for things. Even pkg-config isn't universally used, although I think it's pretty common (at least on Linux).

(I believe modern versions of autoconf try to have configure use pkg-config for a lot of things if it's available, although they can't necessarily do checks of specific APIs without test compiles.)

See also my post: https://leahneukirchen.org/blog/archive/2024/04/what-autoconf-got-right.html

There is another important class of things that configure can test for, which covers optional APIs, like whether or not IPMI libraries are available, and behavior that isn't even exposed as an API (eg, the specific kernel version and its internal behavior).

Some of these seem like misfeatures to me. Kernel versions change after programs are installed, so how can it make sense to check at build time? It doesn't even seem possible when cross-compiling.

And I've been bitten too many times by scripts automatically disabling features because some library wasn't installed when building. Even distributions have had bugs like this, where the packager forgot to list a build dependency, and some feature such as IPMI just disappears on an update. Or appears when I build it myself, because I had an optional package installed for unrelated reasons (and I don't think there's such as thing as a build-anti-dependency).

DESTDIR is an important feature not mentioned yet. One would have to be a little crazy to run "make install" as root; but DESTDIR makes it possible to run that as a normal user, then copy and chown the resulting tree after inspecting it. Much better than --prefix, because most programs shouldn't be hard-coding absolute paths into their binaries anyway.

OpenZFS checks internal kernel APIs because it builds a kernel module and intends to work on a very wide range of kernel versions (and the Linux kernel has no such thing as a stable API). This is imperfect but generally works, even in the face of kernel structures changing fields, kernel functions changing arguments, and what kernel functions to call to do things changing over time.

As far as optional or your choice of libraries goes, Conserver has chosen (for whatever reason) to not require you to build and have the IPMI libraries but instead make IPMI Serial over LAN support optional. Another area of compile time detection of libraries is when there are multiple alternate libraries and some people have reasons to pick one or another. For example, OpenSSL versus GnuTLS for TLS support, which is a choice offered by a number of programs. You could force the person building the software to make a specific choice and then have the build error out if their choice is not present, but this is often considered unfriendly by build configuration systems (and leads to its own errors if you accidentally don't set the option at all).

Detecting optional libraries (and their versions and API levels) would be much easier if everyone just started providing pkgconfig files.

OpenZFS checks internal kernel APIs because it builds a kernel module and intends to work on a very wide range of kernel versions

But that shouldn't involve "detect[ing] various aspects of the system [it's] being built on"; it should be probing the APIs of the system it's being built for. It's kind of important that the module be built for the kernel you intend to boot, especially if that source (or the root filesystem) is stored in OpenZFS.

(I guess your statement is in some sense true, in that the presence of the next-kernel headers is as aspect of the system being built on.)

You could force the person building the software to make a specific choice and then have the build error out if their choice is not present, but this is often considered unfriendly by build configuration systems

I get the user-friendliness aspect, but I value predictability more. Maybe the scripts should have a big "--auto" switch to explicitly ask for as much automation as possible? Distros, of course, shouldn't use that, except perhaps once to generate the "configure" command line to put in the source package. This suggests that, for reproducibility, it'd be good to print such a command line, rather than just the human-readable "detected FOO: yes".

Scripts could also detect that they're being run from a TTY, and if no arguments are present, ask the user whether to auto-detect the optional features.

A lot of Autoconf configure scripts have the opposite of —auto which is called --enable-maintainer-mode and prevents the script from changing build dependencies based on their availability. No autodetection is done to change defaults, they stay in force unless overridden by arguments, and if a dependency is missing the build just fails instead of succeeding by building a lesser-featured artifact.