What I know about Solaris 10 NFS server file lock limits

June 12, 2009

Suppose that you have a Solaris fileserver and a bunch of NFS clients (in this case they're Linux machines, but I don't think it matters). Perhaps one or two of them are Samba servers. Further suppose that your client machines start reporting things like 'fcntl() failed: No locks available'. What is limiting you, and what do you do about it?

(We'll assume that you use tools like lslk to rule out some process on a client machine holding on to a lot of locks.)

Although it's dangerous to extrapolate from current OpenSolaris code (and the lack of documented tuneable parameters), it seems as if Solaris 10 has no intrinsic limit on how many file locks it supports; if you ask for too many for the system to handle, your processes just start going to sleep as they wait for kernel memory to be freed up.

However, there does appear to be a limit on how many concurrent NFS locking requests the server will handle; this is the setting for LOCKD_SERVERS in /etc/default/nfs, which defaults to 20. It appears that if you run into this limit, clients start getting these 'no locks available' messages. Since this is based on concurrent requests, not the total number of locks used, it looks like you can run into this if you merely have some clients doing a lot of short term locking requests.

(It's also possible that you can run into this if the server is short of memory in general, so that the NFS locking requests start running into the 'sleep waiting for kernel memory' issue, clogging up this limited supply of threads.)

From our experience, using svcadm to disable and re-enable the nlockmgr service is enough to pick up changes to /etc/default/nfs. This did make file locking stuff on the clients stall for about thirty seconds or so, but even still that was a lot less drastic than, say, rebooting the fileserver.

Written on 12 June 2009.
« There are two different purposes of monitoring systems
One of the reasons good alerting is tough »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Jun 12 01:28:51 2009
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.