Wandering Thoughts archives

Making tracking upstream Git repositories a bit quieter

I track a bunch of upstream Git repositories, where by 'track' I mean that I keep a local copy and update it periodically. By now I have enough of these tracking repositories that updating them all with a straightforward 'git pull --ff-only' (sometimes through an alias) is increasingly noisy, with too much output. This is especially so for the most active repositories, such as my copy of the Linux kernel; a normal 'git pull' on the Linux kernel can easily produce more than a hundred lines of output (sometimes several hundred). Over the past week I've been trying to hunt around in Git to reduce the amount of normal output, with only some success.

The two obvious steps I've taken are to set the 'git fetch' output to its compact mode with 'git config fetch.output compact' and, for some very active repositories, to turn off diffstat output (which lists files modified, added, and removed) with 'git config merge.stat off'. However, this is still relatively verbose in two areas. First, for all repositories you get the four standard lines of 'remote:' messages that report the progress of the fetch. Second, for some repositories, I get a churn of branches being created and pruned.

As far as I know, there's no way to turn off only the remote progress reporting short of feeding standard output and standard error from 'git pull' (or a direct 'git fetch') through a pipe to, say, cat. You can silence everything with 'git fetch -q' but that's potentially quite extreme; in many repositories I want to know about new tags and new branches because they're important markers of things happening. If I was willing to lose those, I could get a very minimal output with 'get fetch -q && git pull', which produces either 'Already up to date' or two lines of 'Updating ...' and 'Fast-forward'.

Knowing that an update happened is important in some repositories, because I'll then go read the commit messages. In others, it's just noise (although at two lines, tolerable noise). This surfaces the small issue that I'm not actually sure what I want in quiet 'git pull' output, and it's almost certainly different in different repositories. I interact quite differently with my copy of the Linux kernel than I do with, say, my copy of OpenZFS on Linux.

find mostly doesn't need xargs today on modern Unixes

I've been using Unix for long enough that 'find | xargs' is a reflex. When I started and for a long time afterward, xargs was your only choice for efficiently executing a command over a bunch of find results. If you didn't want to run one grep or rm or whatever per file (which was generally reasonably slow in those days), you reached for 'find ... -print | xargs ...'. There were some gotchas in traditional xargs usage, and one of them was why GNU xargs, GNU find, and various other things start growing options to use the null byte as an argument terminator instead of the usual (and surprising) definition. Over time I adopted to these and soon was mostly using 'find ... -print0 | xargs -0 ...'.

For usage with find, all of this is unnecessary on a modern Unix and has been for some time, because find folded this into itself. Modern versions of find don't have just the traditional '-exec', which runs one command per file, but also an augmented version of it which aggregates the arguments together like xargs does. This augmented version is used by ending the '-exec' with '+' instead of ';', like so:

find . ... -exec grep -H whatever '{}' +

(I'm giving grep the -H argument for reasons covered here.)

Although I sometimes still reflexively use 'find | xargs', more and more I'm trying to use the simple form of just find with this augmented -exec. My reflexes can learn new tricks, eventually.

This augmented form of -exec is in the Single Unix Specification for find, so unsurprisingly it's not just in GNU Find but also OpenBSD, FreeBSD, NetBSD, and Illumos. I haven't tried to look up a find manpage in whatever commercial Unixes are left (probably at least macOS and AIX). Based on the rationale section of the SUS find, this very convenient find feature was introduced in System V R4. The Single Unix Specification also explains why they didn't adopt the arguably more Unixy option of '-print0' for null-terminated output.

(In practice everyone has adopted -print0 as well, even OpenBSD and Illumos. I assume without checking that they also all have 'xargs -0', because it doesn't make much sense to adopt one without the other.)

PS: Unfortunately this feature is not quite as flexible as it looks. Both the specification and actual find implementations require the '{}' to be at the end of the command, instead of anywhere in it. This means you can't do something like 'find ... -exec mv {} /some/dir +'. This makes life slightly simpler for find's code and probably only rarely matters for actual usage.