Chris's Wiki :: blog/unix/GroupLimitState

The state of limits on how many groups you can be in (especially for NFS)

May 29, 2014

It is an article of vague Unix folklore that your users can only be in so many groups before they may start having weird problems or things just fail outright. Depending on who you ask and what conditions you're operating under, this limit might be 16 groups, 32 groups, or a lot more. There are actually two limits: the local system limit and a limit imposed by many NFS configurations.

On modern Unixes that you'll want to use, the local limit is generally high. Based on reading manpages and include headers, on current Linux machines it's 64K groups and on current FreeBSD it's 1023 groups. OpenBSD appears to be quite old fashioned with a 16-group restriction. Current Illumos source defaults to a 16-group limit but this can be tuned in /etc/system with 'set ngroups_max=...' if I'm reading the source code correctly.

(See also this resource for other systems and older releases.)

But all of this local optimism is moot in the face of a long standing NFS restriction. Commonly used parts of the NFS protocol are restricted to using 16 groups at most. This limit appeared in the original NFS but it has not been raised in NFS v3 or even, sort of, in NFS v4. More specifically, this limitation is part of the default 'plain Unix permissions' authentication, called AUTH_SYS, that is normally used in NFS v3 and may still be used in NFS v4. In theory you can switch NFS v3 to use something else, but in practice Mike Eisler notes that NLM locking may still cause you problems (that article is a good background in general and discusses potential workarounds).

If you have Linux NFS servers there is a potential workaround where you can make the server ignore the list of groups that the clients send and look up group information locally. There is still a size limit but it's much larger. See Kyle Anderson's Solving the NFS 16-Group Limit Problem for the details. I haven't tested this, partly because our NFS servers don't run Linux so I can't use it.

So, in short, we're going to keep bumping into the 16-group limit every so often for some people here and there is no good workaround for us.

(Every so often I set out to investigate something and wind up finding out that there is no good news and things are exactly as my vague memory of folklore thought they were. Today is one of those cases.)

Comments on this page:

By Ewen McNeill at 2014-05-29 05:36:12:

Older Linux (2.4 kernels and earlier IIRC) had a local limit of 32 groups. I remember this from a RHEL 3 system I was looking after which was running Samba and joined to the Windows domain. We ended up manually patching each RPM update of Samba to "use the first 32 groups" rather than using no groups if there were more than 32 available -- and RedHat wouldn't take the patch into RHEL 3. (It wasn't needed later because the rest are based on 2.6+ kernels which have many more groups.)

I mention this mostly because if you have the source to the client end you may be able to make your own work around. For our scenario it happened to work out that the few groups (of the 35-ish on some users) that the Samba server needed to care about fell into the first 32 (up through when the server was retired). But in your case it might be that you could somehow teach the client to send the appropriate 16 groups to each NFS server. I'm thinking, perhaps, some sort of config file which had a per-server list of "important groups" to send, and the client code could prioritise those ones and then include any others if there was space. Then you'd be out of luck only if any given server needed to know about more than 16 groups. (In which case maybe you could partition by NFS server IP, and have two IPs on one physical host?)

Ewen

PS: I don't remember the technical detail of NFS well enough to know if you can get a bonus 17th group by using the "primary group" (eg, /etc/passwd) and "supplementary groups" (eg, /etc/groups) trick. Maybe not given the link you provide. (It didn't help with our Samba problem as all the "important" groups came from the Windows domain.)

By David Magda at 2014-05-29 16:30:28:

For Solaris, the functionality to have the server do the look-ups is only available in 11.1:

http://tinyurl.com/qyhjm3q

http://www.c0t0d0s0.org/archives/7634-Less-known-Solaris-11.1-features-A-user-in-1024-groups-and-a-workaround-for-a-25-year-old-problem.html

ngroups_max does indeed change things locally on (Open)Solaris:

https://blogs.oracle.com/peteh/entry/increasing_unix_group_membership_easy

Written on 29 May 2014.

«	Yahoo Groups has a bad spam problem and they don't care

In Go, sometimes a nil is not a nil »