2020-09-30
People still use newgrp
(to my surprise)
One of the things that came out of me writing about why newgrp
exists and casually mentioning that it was a
relic (and that OpenBSD had gotten rid of it) is that a number of
people left comments discussing how they actively use newgrp
even
today. This surprised me more than a bit, although in retrospect it
probably shouldn't have.
There's long been an aphorism in software development that if you have a large and complicated program (such as a word processor), 80% of the people only use 20% of its features but they all use a slightly different 20% (and so if you make a simple version with 20% of the features, many more people will be uninterested in it than you expect). Unix is not a single program, but it is large and complicated and so it's subject to much the same dynamics. I may use only 20% of the programs on my Unix and you may use only 20% of the ones on yours, but our collections probably don't entirely overlap.
(Sometimes this surfaces in those 'share your top N most frequently used shell commands' threads, where you'll find a surprising diversity in all the answers.)
The corollary to this is that you can find some people (still) using
almost any somewhat obscure Unix command with useful features, and
of course that includes newgrp
. This is true even if you solve
the problems they use the command for in entirely different ways
(and even if your way is probably objectively better; sometimes
it's not worth the bother of learning new things).
Among other things, people's use of Unix commands is influenced by
their history with Unix, and everyone has taken a somewhat different
path there.
(I can and do say various things about OpenBSD, but one of their
positive sides is their anti-fossilization stance. OpenBSD will
remove things, like newgrp
, even if people are still using them.
So will FreeBSD, as it turns out, since in a default FreeBSD
installation newgrp
is not setuid and as a result doesn't do
anything unless you're root.)
Sidebar: What I use instead of newgrp
The two big uses for newgrp
that came up in comments on the
original entry were taking on newly added group
permissions (when you've just been added to a group in /etc/group
)
without logging out and then back in again, and insuring that new
files get created in the right group.
The latter is the easy case. Any time that we actually care what group new files and subdirectories wind up in on our Linux systems, we make the directory setgid so that everything under it inherits whatever group it has. As I discovered before, on FreeBSD you don't have a choice, and this traditional BSD behavior actually makes plenty of sense.
(I haven't checked OpenBSD but I suspect it behaves the same way as FreeBSD.)
My less satisfying answer is for the the 'just added to a new group'
case. This doesn't come up for me very often, and when it does I
do wind up logging out and then back in again. I may defer having
to do a session restart for a while by various mechanisms like
sshing to my machine again or using 'su --login
', but eventually
there are enough irritations that I want to do a session restart.
For instance, things started from my window manager (including new
terminal windows) won't be in the new group, so I'll have to keep
doing whatever it is to access the group.
(It helps that I have gotten session restarts to be relatively low annoyance things, partly in self defense.)
Using DMARC information is complicated in practice in the real world
As part of a planned switch to rspamd as our anti-spam system (well, our spam recognition system), I've been taking a closer look at how our test rspamd scores some email and what it reports about why. This has given me a new and unhappy view of DMARC in the real world, building on how DKIM looks for our 'good' email. So let me tell you a story, starting with the background.
The university is now a big user of Microsoft Teams. The university's UTmail+ institutional email system for staff, faculty, and so on is also "powered by Office 365", which is to say that it is hosted in Microsoft's 'Outlook/Office 365' cloud (with a bunch of contractual terms around what datacenters our data actually lives in and so on). However, while everyone here has a UTmail+ account (which is also your Teams account), some people forward their email to other systems. In particular, any number of people here forward their UTmail+ email to us.
Various activity in Microsoft Teams generates email to you, which normally is sent to your institutional email account in UTmail+ and may then get forwarded to us. As a good email citizen in the modern world, Microsoft Teams DKIM signs its email:
DKIM: d=email.teams.microsoft.com s=selector1 c=relaxed/relaxed a=rsa-sha256
Such email comes from 'noreply@email.teams.microsoft.com' as both the envelope sender and the From: header address.
As a good email citizen in the modern world, the Outlook/Office 365 environment also DKIM signs outgoing email, this time under the 'onmicrosoft.com' domain instead of microsoft.com:
DKIM: d=utoronto.onmicrosoft.com s=[...] c=relaxed/relaxed a=rsa-sha256
Unfortunately, when Microsoft Teams email transits either our hosted Office 365 environment or at least one other institution's hosted one, something breaks the Microsoft Teams DKIM signature. At the same time, Office 365 does not change the From:, which means that the message is covered by whatever DMARC policy applies to that 'noreply@email.teams' address.
The email.teams.microsoft.com sub-subdomain does not have its own DMARC policy. Instead it falls under the general microsoft.com DMARC policy:
v=DMARC1; p=reject; pct=100; [...]
This says that anything with a From: of a microsoft.com address that fails DMARC and SPF should be rejected. Since Microsoft advertises this DMARC policy, rspamd takes them at their word and applies a spam score penalty to all of this entirely legitimate email.
There is a standard that is supposed to deal with this problem, ARC. The Office 365 forwarding environment appears to put on various ARC headers, but in a sample Teams email I have these headers seem to claim not to have validated the original DKIM signature, just the SPF results. In any case, it feels like there would need to be some explicit configuration somewhere so that either we or Microsoft would give ARC signatures from these Office 365 environments the power to give a pass on microsoft.com's DMARC policy. In practice rspamd sometimes applies a mild bonus (ie, not-spam) for ARC_ALLOW, and sometimes gives it a neutral result of 'no change in score' (a 0.00 result, at least according to the logs).
(It appears that rspamd 2.5 adds an undocumented option to its ARC module to configure a whitelist of trusted ARC signers, per here, here and here, but without documentation I'm wary of touching it even if I could figure out what exactly to put where.)
Beyond the specific problem here and the opacity of what is going on in evaluating DKIM, DMARC, and ARC results in rspamd, notice how we have wound up in a situation where none of these things can be checked in isolation. In practice, you cannot just look at the From: domain and see whether it passes DMARC and DKIM checks; you need additional validation that is conditional on, well, something. In practice, to reliably accept Microsoft Teams email that transits a hosted Office 365 environment we would probably have to identify all of the outgoing sources for this, including for people who forward their Microsoft Teams email through another place that hosts their email in Office 365.
The practical result of all of this is that today I reconfigured rspamd to not assign any spam score penalty for a DMARC failure. DMARC failures are clearly going to happen with legitimate email to our users that transits through their UTmail+ addresses (at least), which in practice makes them useless.
How the Unix newgrp
command behaved back in V7 Unix
In why newgrp
exists (sort of), we discovered
that Unix groups can have passwords, that there is a newgrp
command
that lets you change your (primary) group, and that it dates back
to V6 Unix, much earlier than I expected, and that how the V7
newgrp
command behaved was interesting. V7 Unix source code (and
manual pages and more) is online through tuhs.org, so we can actually read the V7 newgrp.c
source code directly.
(We can also read the V6 source code, but it's much more complicated so I'm going to use V7. Besides, V7 is what everyone generally thinks of as the birth point of modern Unix; everything before V7 is at least a bit odd and different.)
How I expected newgrp
to behave with password protected groups
is that anyone could say 'newgrp GROUP
', and if they knew the
password they were put into the group without having to be listed
as a (potential) member in /etc/group
. This may be how your local
Unix's newgrp
command works today, but it is not how the V7 newgrp
did. Let's start with an extract from the newgrp.1
manpage:
A password is demanded if the group has a password and the user himself does not.
When most users log in, they are members of the group named `other.' Newgrp is known to the shell, which executes it directly without a fork.
First, unless you were newgrp
ing to the special group "other",
you had to be listed in the group's /etc/group
entry whether or
not the group had a password. Then, as the manpage describes, a
group password was only enforced if your own Unix account didn't
have one. If your account did have a password, the group password
was ignored. In other words, you could not use group passwords to
give everyone potential access to a group if they knew its group
password; the group password was only used to protect groups from
unpassworded, open access logins (if you had any, and if you listed
them in an additional group).
When dealing with groups with passwords, the normal Linux version
of newgrp
retains the V7 behavior for password protected groups
that you're listed as a member of, but also allows you to newgrp
to any group with a group password if you know it (according to
its manpage,
I haven't tested it). The FreeBSD version I have access to specifically
discourages use of group passwords and says that newgrp
is usually
installed without the setuid bit (cf); it claims to only ask
for a password if you're not listed as a member of the group.
(I believe this means that in a default setup the FreeBSD newgrp
can't
be used to change yourself into a group that you've just been added to
in /etc/group
.)
You might wonder about the last sentence of the V7 manpage bit that
I quoted. The reason for this particular hack is to not leave an
extra shell process around that you probably don't want (and that
would be using up extra resources on the small machines that V7 ran
on). Conceptually, if you're running newgrp
you generally want
your current shell session to be switched to the new group. If the
shell just ran newgrp
as a regular command, it would switch groups
for its own process and then exec /bin/sh
to give you a new shell
in the new group; it would be like you'd run sh
in your login
shell. If you repeatedly newgrp
'd, you'd wind up with a whole
stack of shells.
So instead the V7 Bourne shell specially recognizes newgrp
(along
with login
) and doesn't fork to run it; instead it directly exec
s
into newgrp
, replacing your login shell with newgrp
(which will
then ideally replace it with another shell, and indeed the V7
newgrp
tries to always wind up exec'ing a new shell).
(This magic is hiding in the combination of msg.c
and xec.c;
see the handling of SYSLOGIN
.)