As it turns out, there are some limitations with our custom NFS mount authorization hack that we've discovered. First, Solaris 10 puts some security restrictions on what mountd can do; the specific bit that we ran into is it can't fork() and exec() shell scripts. Mountd can make outgoing socket connections, so we got around this by putting most of the complex work in an inetd-spawned 'daemon'. (There is probably a way to turn this security feature off, but we didn't bother trying to find it.)

Second, things won't stall endlessly waiting for an answer to an NFS mount authorization. I don't know how long the various timeouts are, but there is definitely a limit on how much time your authorization code can use. This probably won't be a real limit in most environments, since the timeout seems to be at least several seconds, but bear it in mind.

We've also seen indications that mountd will only allow a certain (low) number of outstanding mount requests; my current guess is 20. Once it hits the limit, further requests are either dropped or get permission denied errors (it's not clear which, and we've lacked the time to debug things in detail when the issue has come up). This is also going to be an issue if your authorization code can take some time.

Finally, it seems that under some circumstances some part of the overall system will cache the answers to mount requests, with the effect that that an authorization decision can be 'sticky' for some amount of time even if re-running the innetgr() query from scratch would get a different answer. It's possible that this is mostly for permission denied answer, as spurious mount denials are when we notice this.

(Thus, this is probably not the right hack for you if your code needs to take a shot at absolutely every mount request mountd handles, no matter what.)

All of this is inevitable given that the entire thing is a hack; one can't expect the rest of the system to cooperate perfectly, the way one could if this was a real feature. It follows that the more your code deviates from normal innetgr() behavior, the more possibility there is for something to go wrong. If you need a seriously different NFS mount authorization scheme, you may have to hack the OpenSolaris mountd code base directly (and hope that it works okay on regular Solaris 10, which it may not).