Sometimes the way to solve a problem is to rethink the problem
After a certain amount of exploration and discussion, we've come up with what we feel is a solid solution for getting our NFS mount authentication working on Linux. Our solution is to not use Linux; instead we'll use OmniOS, where we already have a perfectly working NFS mount authentication system.
To get there we had to take a step back and look at our actual objectives and constraints. The reason we wanted our NFS mount authentication on Linux is that we want to offer a service where people give us disks (plus some money for overhead) and we put them into something and make them available via NFS and Samba and so on. The people involved very definitely want their disks pace available via NFS because they want their disk space to be conveniently usable (and fast) from various existing Linux compute machines and so on. We wanted to do this on Linux (as opposed to OmniOS (or FreeBSD)) because we trust Linux's disk drivers the most and in fact we already have Linux running happily on 16-bay and 24-bay SuperMicro chassis.
(I did some reading and experimentation with OmniOS management of LSI SAS devices and was not terribly enthused by it.)
We haven't changed our minds about using Linux instead of OmniOS to talk to the disks; we've just come to the blindingly obvious realization that we've already solved this problem and all it takes to reduce our current situation to our canned solution is adding a server running OmniOS in front of the Linux machine with the actual disks. Since we don't view this bulk disk hosting as an critical service and it doesn't need 10G Ethernet (even if that worked for us right now), this second server can be one of our standard inexpensive 1U servers that we have lying around (partly because we tend to buy in bulk when we have some money).
(Our first round implementation can even take advantage of existing hardware; since we're starting to decommission our old fileserver environment we have both spare servers and more importantly spare disk enclosures. These won't last forever, but they should last long enough to find out if there's enough interest in this service for us to buy 24-bay SuperMicro systems to be the disk hosts.)
This rethinking of the problem is not as cool and interesting as, say, writing a Go daemon to do efficient bulk authentication of machines and manage Linux iptables permissions to allow them NFS access, but it solves the problem and that's the important thing. And we wouldn't have come up with our solution if we'd stayed narrowly focused on the obvious problem in front of us, the problem of NFS mount authentication on Linux. Only when one of my coworkers stepped back and started from the underlying problem did we pivot to 'is there any reason we can't throw hardware at the problem?'.
There is a valuable lesson for me here. I just hope I remember it for the next time around.