Some git repository manipulations that I don't know how to do well
For a while now, I've been both tracking upstream git repositories and carrying and rebasing local changes on top of some of them in a straightforward way. The rebasing has been pretty easy and a clear win. But recently I've run into a number of more complicated cases where I'm not sure that I'm using git in the best way. So here are a collection of problems that I'm semi-solving.
I keep local tracking copies of a few upstream repos that are rebased
periodically, such as this one.
I have no local changes in such tracking repositories (ie my local
master is always the same as
origin/master). When the upstream
rebases my plain
git pull will fail, either telling me that it
can't be fast-forwarded or demanding that I merge things. In the
past I've just deleted the entire repo and re-cloned as the simple
way out. Recently I've constructed the manual fix:
git pull --ff-only [fails, but it's fetched stuff] git checkout -f -B master origin/master
It would be nice to have the repo set up so that a plain '
pull' would do this, ideally only if it's safe. I could script
this but there are only some repos that I want to do this for; for
others, either I have local changes or this should never happen.
git pull manpage has some interesting wording that makes it
sound like asking for a rebase here will maybe do the right thing.
But I just found that and contriving a test case is not trivial.
Or maybe I want '
git pull -f'. And now that I've done some web
searches, apparently I want '
git reset --hard origin/master'.
Git is like Perl; sometimes it has so many different paths to the
Next, I'm tracking the repo for my favorite RAW development software and have some little local fixes added on top. Normally I build from the latest git tip, but that's sort of in flux right now so I want to switch to the release branch but still add my changes on top. I'm doing this like so:
git checkout darktable-2.0.x git cherry-pick origin/master..master
I think that using
git cherry-pick here instead of some form of
git rebase is probably the correct approach and this is the one
case where I'm doing things right.
I'm tracking the main repository for my shell and applying my changes on top of
it. However, there is a repository of interesting changes that I want to try out; of course
I still want my local changes on top of this. When I did this I
think what I did was '
git pull --rebase /the/other/repo', but I
believe that probably wasn't the right approach. I suspect that
what I really wanted to do was add the second repo as an alternate
upstream, switch to it, and either cherry-pick or rebase my own
changes on top.
Except, well, looking back at my notes about working with Github PRs it strikes me that this is basically the same situation (except this repo isn't an explicit PR). I should probably follow the same process instead of hacking my way around the way I did this time.
Finally, we combine the two situations: I'm building on top of the repo of interesting changes and it rebases itself. Now I want to replace the old version with the new version but reapply my changes on top. I'm not going to try to write down the process I used this time, because I'm convinced it's not optimal; I basically combined the reset origin plus cherry-pick process, using explicit commit hashes for the cherry-picking and recording them beforehand. Based on Aristotle Pagaltzis's comment on my Github PR entry, I think I want something from this Stackoverflow Q&A but I need to read the answers carefully and try it out before I'm sure I understand it all. It does look relatively simple.
(This writeup of
git rebase --onto is helpful too,
but again I need to digest it. Probably I need to redo this whole
procedure the right way the next time this comes up, starting from
a properly constructed repo.)
Sidebar: How I test this stuff out
I always, always test anything like this on a copy of my repo, not
on the main version. I make these copies with
making repo copies with '
git clone' changes how git sees things
like upstreams (for obvious reasons). I suspect that this is the
correct approach and there is no magic '
git clone' option that
does what I want here.
In search of modest scale structured syslog analysis
Every general issue should start from a motivating usage case, so here's ours: we want to be able to find users who haven't logged with SSH or used IMAP in the past N months (this perhaps should include Samba authentication as well). As a university department that deals with graduate students, postdocs, visiting researchers, and various other sorts of ongoing or sporadic collaborations, we have a user population that's essentially impossible to keep track of centrally (and sometimes at all). So we want to be able to tell people things like 'this account that you sponsor doesn't seem to have been used for a year'.
As far as I can tell from Internet searches and so on, there are an assorted bunch of log aggregation, analysis, and querying tools. Logstash is the big one that many people have heard of, but then there's Graylog and fluentd and no doubt others. In theory any of these ought to be the solution to our issue. In practice, there seem to be two main drawbacks:
- They all seem to be designed for large to very large environments.
We have what I tend to call a midsized environment;
what's relevant here is that we only have on the order of 20 to
30 servers. Systems designed for large environments seem to be
both complicated and heavyweight, requiring things like JVMs and
multiple servers and so on.
- None of them appear to come with or have a comprehensive set of parsers to turn syslog messages from various common programs into the sort of structured information that these systems seem designed to work with. You can write your own parsers (usually with regular expressions), but doing that well requires a relatively deep knowledge of just what messages the programs can produce.
(In general all of these systems feel as if they're primarily focused on application level logging of structured information, where you have your website or backend processing system or whatever emit structured messages into the logging backend. Or perhaps I don't understand how you're supposed to use these systems.)
We can undoubtedly make these systems solve our problem. We can set
up the required collection of servers and services and get them
talking to each other (and our central syslog server),
and we can write a bunch of grok patterns
to crack apart
sshd and Dovecot and Samba messages. But all of
this feels as if we are using the back of a large and very sharp
axe to hammer in a small nail. It works, awkwardly, but it's probably
not the right way.
It certainly feels as if structured capturing and analysis of syslog
messages from common programs like
sshd, Dovecot, and so on in a
moderate sized environment ought to be a well solved problem. We
can't be the first people to want to do this, so this particular
wheel must have been reinvented repeatedly by now. But I can't find
even a collection of syslog parsing patterns for common Unix daemons,
much less a full system for this.
(If people know of systems or resources for doing this, we would of course be quite interested. There are some SaaS services that do log analysis for you, but as a university department we're not in a position to pay for this (staff time is free, as always).)