A kernel NFS error message explained

Suppose that your machine is an NFS client, and it periodically logs a kernel error message that looks like:

nfs_stat_to_errno: bad nfs status return value: 16

What does this mean?

The short summary is that your NFS server is violating the NFS specifications. NFS requests can fail, and when they do the server returns an error number to tell the client why. The NFS specs say what the allowed error numbers are; the Linux kernel client code checks that the error number it got was one of the allowed ones. You get this message when this check fails. The number is the (decimal) number of the invalid NFS error code.

(You can find the list of valid NFS v3 error codes in RFC 1813 section 2.5 and the NFS v4 error codes in RFC 3530 section 18. I believe that NFS v2 error codes are a subset of NFS v3 ones, and NFS v4 ones are almost a superset of NFS v3 ones.)

Now some guesses as to what those stray error values actually mean.

If you look at the actual numbers of those defined error values and have a Solaris machine handy (or just a general knowledge of Unix), something jumps out at you immediately: many of the NFS 'error codes' actually have the same numbers as the corresponding SunOS errno value for the problem. I strongly suspect that the original NFS server and client code did not have NFS 'error codes' as such; instead it took the server's kernel errno value generated from trying to do the request, stuffed it in the NFS reply, and on the client took the NFS error code and set errno to that.

I further suspect that some NFS servers still do this. Thus, if you get get such an error message from your client kernels, your best bet at figuring out what the NFS server is trying to say is to look at what error that value is on your NFS server. Figuring out why some operation seems to be getting this error is up to you.

(In the specific case of this message for us, our NFS servers are Solaris 10 machines, where 16 is EBUSY. Low errno numbers are relatively well standardized; high ones are not necessarily so.)