Over on the Fediverse, I said something:

I hope that the Grub developers will someday fix grub-install so that the "unknown filesystem" error is replaced with a better one, like "Grub doesn't have the driver(s) necessary to use your / (or /boot) filesystem" or even "Grub doesn't currently support some filesystem features that are enabled on your / (or /boot) filesystem". Ideally with the right filesystem name.

This has certainly been coming up and getting forum/etc answers for long enough. But alas.

Actually fixing the message to be accurate is difficult because of how Grub's code is structured. The simplest improvement is to change the text of the message to "unknown filesystem or filesystem with unsupported features", which at least hints at the potential issue (although the message would have to be re-translated into various languages and so on, so perhaps the Grub developers would be unenthused).

This message can be produced either by grub-install, running on a booted system, or by the Grub bootloader code itself, as you boot the system. Normally it's seen when you run grub-install, which is somewhat puzzling; how is the filesystem unknown when the kernel is using it? And why does grub-install care?

When Grub is booting your system, it doesn't (and can't) use the Linux kernel's filesystem code and device drivers (or any Unix kernel's code; Grub runs in non-Linux environments as well). At the same time, Grub wants to read various things from your filesystems, such as its menu file or your kernel (and on Linux, initramfs). To do this, Grub has its own collection of filesystem code and software disk drivers, generally in a collection of loadable (Grub) modules. When grub-install runs, one of its jobs is to prepare the set of filesystem and disk driver modules Grub will need at boot time. Its report of "unknown filesystem" means that it can't find a filesystem module that will accept the filesystem that you have things on (generally either the root filesystem or your /boot filesystem, depending on whether /boot is on its own filesystem).

The specific message is generated in grub_fs_probe() in kern/fs.c. This function is handed a 'grub device' and runs through grub's list of known filesystem modules, asking each one of them in turn if they can handle the filesystem on the 'grub device'. Currently, filesystem modules return the same error code if the device isn't their type of filesystem or if it's their type of filesystem but it has filesystem features that the Grub module doesn't (yet) support. The filesystem module can set a specific error message here (in addition to its error code), but grub_fs_probe() doesn't normally report the per filesystem error messages unless (the right sort of) debugging is turned on (this can be done in grub-install with '-vv', although that enables all debugging messages and produces a lot of messages). Instead, if all filesystem modules say they can't handle the filesystem, grub_fs_probe() reports a generic "unknown filesystem" error. One level up, grub-install.c calls grub_fs_probe() (in a couple of different places) and then reports the error message that it's produced (if it failed).

Fixing this to return an exact error message about what's wrong is at least a little bit tricky and would make the code more complicated. It also touches a relatively critical piece of Grub, since this code is also run during boot (and must properly accept the filesystem then). So I suspect the most that Grub developers would do is change the message to a longer version that mentions the possibility of feature flag mismatches.