In why newgrp exists (sort of), we discovered that Unix groups can have passwords, that there is a newgrp command that lets you change your (primary) group, and that it dates back to V6 Unix, much earlier than I expected, and that how the V7 newgrp command behaved was interesting. V7 Unix source code (and manual pages and more) is online through tuhs.org, so we can actually read the V7 newgrp.c source code directly.

(We can also read the V6 source code, but it's much more complicated so I'm going to use V7. Besides, V7 is what everyone generally thinks of as the birth point of modern Unix; everything before V7 is at least a bit odd and different.)

How I expected newgrp to behave with password protected groups is that anyone could say 'newgrp GROUP', and if they knew the password they were put into the group without having to be listed as a (potential) member in /etc/group. This may be how your local Unix's newgrp command works today, but it is not how the V7 newgrp did. Let's start with an extract from the newgrp.1 manpage:

A password is demanded if the group has a password and the user himself does not.

When most users log in, they are members of the group named `other.' Newgrp is known to the shell, which executes it directly without a fork.

First, unless you were newgrping to the special group "other", you had to be listed in the group's /etc/group entry whether or not the group had a password. Then, as the manpage describes, a group password was only enforced if your own Unix account didn't have one. If your account did have a password, the group password was ignored. In other words, you could not use group passwords to give everyone potential access to a group if they knew its group password; the group password was only used to protect groups from unpassworded, open access logins (if you had any, and if you listed them in an additional group).

When dealing with groups with passwords, the normal Linux version of newgrp retains the V7 behavior for password protected groups that you're listed as a member of, but also allows you to newgrp to any group with a group password if you know it (according to its manpage, I haven't tested it). The FreeBSD version I have access to specifically discourages use of group passwords and says that newgrp is usually installed without the setuid bit (cf); it claims to only ask for a password if you're not listed as a member of the group.

(I believe this means that in a default setup the FreeBSD newgrp can't be used to change yourself into a group that you've just been added to in /etc/group.)

You might wonder about the last sentence of the V7 manpage bit that I quoted. The reason for this particular hack is to not leave an extra shell process around that you probably don't want (and that would be using up extra resources on the small machines that V7 ran on). Conceptually, if you're running newgrp you generally want your current shell session to be switched to the new group. If the shell just ran newgrp as a regular command, it would switch groups for its own process and then exec /bin/sh to give you a new shell in the new group; it would be like you'd run sh in your login shell. If you repeatedly newgrp'd, you'd wind up with a whole stack of shells.

So instead the V7 Bourne shell specially recognizes newgrp (along with login) and doesn't fork to run it; instead it directly execs into newgrp, replacing your login shell with newgrp (which will then ideally replace it with another shell, and indeed the V7 newgrp tries to always wind up exec'ing a new shell).

(This magic is hiding in the combination of msg.c and xec.c; see the handling of SYSLOGIN.)