Wandering Thoughts archives


People still use newgrp (to my surprise)

One of the things that came out of me writing about why newgrp exists and casually mentioning that it was a relic (and that OpenBSD had gotten rid of it) is that a number of people left comments discussing how they actively use newgrp even today. This surprised me more than a bit, although in retrospect it probably shouldn't have.

There's long been an aphorism in software development that if you have a large and complicated program (such as a word processor), 80% of the people only use 20% of its features but they all use a slightly different 20% (and so if you make a simple version with 20% of the features, many more people will be uninterested in it than you expect). Unix is not a single program, but it is large and complicated and so it's subject to much the same dynamics. I may use only 20% of the programs on my Unix and you may use only 20% of the ones on yours, but our collections probably don't entirely overlap.

(Sometimes this surfaces in those 'share your top N most frequently used shell commands' threads, where you'll find a surprising diversity in all the answers.)

The corollary to this is that you can find some people (still) using almost any somewhat obscure Unix command with useful features, and of course that includes newgrp. This is true even if you solve the problems they use the command for in entirely different ways (and even if your way is probably objectively better; sometimes it's not worth the bother of learning new things). Among other things, people's use of Unix commands is influenced by their history with Unix, and everyone has taken a somewhat different path there.

(I can and do say various things about OpenBSD, but one of their positive sides is their anti-fossilization stance. OpenBSD will remove things, like newgrp, even if people are still using them. So will FreeBSD, as it turns out, since in a default FreeBSD installation newgrp is not setuid and as a result doesn't do anything unless you're root.)

Sidebar: What I use instead of newgrp

The two big uses for newgrp that came up in comments on the original entry were taking on newly added group permissions (when you've just been added to a group in /etc/group) without logging out and then back in again, and insuring that new files get created in the right group.

The latter is the easy case. Any time that we actually care what group new files and subdirectories wind up in on our Linux systems, we make the directory setgid so that everything under it inherits whatever group it has. As I discovered before, on FreeBSD you don't have a choice, and this traditional BSD behavior actually makes plenty of sense.

(I haven't checked OpenBSD but I suspect it behaves the same way as FreeBSD.)

My less satisfying answer is for the the 'just added to a new group' case. This doesn't come up for me very often, and when it does I do wind up logging out and then back in again. I may defer having to do a session restart for a while by various mechanisms like sshing to my machine again or using 'su --login', but eventually there are enough irritations that I want to do a session restart. For instance, things started from my window manager (including new terminal windows) won't be in the new group, so I'll have to keep doing whatever it is to access the group.

(It helps that I have gotten session restarts to be relatively low annoyance things, partly in self defense.)

unix/NewgrpSurprisingUsage written at 23:29:20; Add Comment

Using DMARC information is complicated in practice in the real world

As part of a planned switch to rspamd as our anti-spam system (well, our spam recognition system), I've been taking a closer look at how our test rspamd scores some email and what it reports about why. This has given me a new and unhappy view of DMARC in the real world, building on how DKIM looks for our 'good' email. So let me tell you a story, starting with the background.

The university is now a big user of Microsoft Teams. The university's UTmail+ institutional email system for staff, faculty, and so on is also "powered by Office 365", which is to say that it is hosted in Microsoft's 'Outlook/Office 365' cloud (with a bunch of contractual terms around what datacenters our data actually lives in and so on). However, while everyone here has a UTmail+ account (which is also your Teams account), some people forward their email to other systems. In particular, any number of people here forward their UTmail+ email to us.

Various activity in Microsoft Teams generates email to you, which normally is sent to your institutional email account in UTmail+ and may then get forwarded to us. As a good email citizen in the modern world, Microsoft Teams DKIM signs its email:

DKIM: d=email.teams.microsoft.com s=selector1 c=relaxed/relaxed a=rsa-sha256

Such email comes from 'noreply@email.teams.microsoft.com' as both the envelope sender and the From: header address.

As a good email citizen in the modern world, the Outlook/Office 365 environment also DKIM signs outgoing email, this time under the 'onmicrosoft.com' domain instead of microsoft.com:

DKIM: d=utoronto.onmicrosoft.com s=[...] c=relaxed/relaxed a=rsa-sha256

Unfortunately, when Microsoft Teams email transits either our hosted Office 365 environment or at least one other institution's hosted one, something breaks the Microsoft Teams DKIM signature. At the same time, Office 365 does not change the From:, which means that the message is covered by whatever DMARC policy applies to that 'noreply@email.teams' address.

The email.teams.microsoft.com sub-subdomain does not have its own DMARC policy. Instead it falls under the general microsoft.com DMARC policy:

v=DMARC1; p=reject; pct=100; [...]

This says that anything with a From: of a microsoft.com address that fails DMARC and SPF should be rejected. Since Microsoft advertises this DMARC policy, rspamd takes them at their word and applies a spam score penalty to all of this entirely legitimate email.

There is a standard that is supposed to deal with this problem, ARC. The Office 365 forwarding environment appears to put on various ARC headers, but in a sample Teams email I have these headers seem to claim not to have validated the original DKIM signature, just the SPF results. In any case, it feels like there would need to be some explicit configuration somewhere so that either we or Microsoft would give ARC signatures from these Office 365 environments the power to give a pass on microsoft.com's DMARC policy. In practice rspamd sometimes applies a mild bonus (ie, not-spam) for ARC_ALLOW, and sometimes gives it a neutral result of 'no change in score' (a 0.00 result, at least according to the logs).

(It appears that rspamd 2.5 adds an undocumented option to its ARC module to configure a whitelist of trusted ARC signers, per here, here and here, but without documentation I'm wary of touching it even if I could figure out what exactly to put where.)

Beyond the specific problem here and the opacity of what is going on in evaluating DKIM, DMARC, and ARC results in rspamd, notice how we have wound up in a situation where none of these things can be checked in isolation. In practice, you cannot just look at the From: domain and see whether it passes DMARC and DKIM checks; you need additional validation that is conditional on, well, something. In practice, to reliably accept Microsoft Teams email that transits a hosted Office 365 environment we would probably have to identify all of the outgoing sources for this, including for people who forward their Microsoft Teams email through another place that hosts their email in Office 365.

The practical result of all of this is that today I reconfigured rspamd to not assign any spam score penalty for a DMARC failure. DMARC failures are clearly going to happen with legitimate email to our users that transits through their UTmail+ addresses (at least), which in practice makes them useless.

spam/DMARCPracticalComplexity written at 22:10:40; Add Comment

How the Unix newgrp command behaved back in V7 Unix

In why newgrp exists (sort of), we discovered that Unix groups can have passwords, that there is a newgrp command that lets you change your (primary) group, and that it dates back to V6 Unix, much earlier than I expected, and that how the V7 newgrp command behaved was interesting. V7 Unix source code (and manual pages and more) is online through tuhs.org, so we can actually read the V7 newgrp.c source code directly.

(We can also read the V6 source code, but it's much more complicated so I'm going to use V7. Besides, V7 is what everyone generally thinks of as the birth point of modern Unix; everything before V7 is at least a bit odd and different.)

How I expected newgrp to behave with password protected groups is that anyone could say 'newgrp GROUP', and if they knew the password they were put into the group without having to be listed as a (potential) member in /etc/group. This may be how your local Unix's newgrp command works today, but it is not how the V7 newgrp did. Let's start with an extract from the newgrp.1 manpage:

A password is demanded if the group has a password and the user himself does not.

When most users log in, they are members of the group named `other.' Newgrp is known to the shell, which executes it directly without a fork.

First, unless you were newgrping to the special group "other", you had to be listed in the group's /etc/group entry whether or not the group had a password. Then, as the manpage describes, a group password was only enforced if your own Unix account didn't have one. If your account did have a password, the group password was ignored. In other words, you could not use group passwords to give everyone potential access to a group if they knew its group password; the group password was only used to protect groups from unpassworded, open access logins (if you had any, and if you listed them in an additional group).

When dealing with groups with passwords, the normal Linux version of newgrp retains the V7 behavior for password protected groups that you're listed as a member of, but also allows you to newgrp to any group with a group password if you know it (according to its manpage, I haven't tested it). The FreeBSD version I have access to specifically discourages use of group passwords and says that newgrp is usually installed without the setuid bit (cf); it claims to only ask for a password if you're not listed as a member of the group.

(I believe this means that in a default setup the FreeBSD newgrp can't be used to change yourself into a group that you've just been added to in /etc/group.)

You might wonder about the last sentence of the V7 manpage bit that I quoted. The reason for this particular hack is to not leave an extra shell process around that you probably don't want (and that would be using up extra resources on the small machines that V7 ran on). Conceptually, if you're running newgrp you generally want your current shell session to be switched to the new group. If the shell just ran newgrp as a regular command, it would switch groups for its own process and then exec /bin/sh to give you a new shell in the new group; it would be like you'd run sh in your login shell. If you repeatedly newgrp'd, you'd wind up with a whole stack of shells.

So instead the V7 Bourne shell specially recognizes newgrp (along with login) and doesn't fork to run it; instead it directly execs into newgrp, replacing your login shell with newgrp (which will then ideally replace it with another shell, and indeed the V7 newgrp tries to always wind up exec'ing a new shell).

(This magic is hiding in the combination of msg.c and xec.c; see the handling of SYSLOGIN.)

unix/NewgrpV7Behavior written at 00:40:50; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.