In search of modest scale structured syslog analysis
Every general issue should start from a motivating usage case, so here's ours: we want to be able to find users who haven't logged with SSH or used IMAP in the past N months (this perhaps should include Samba authentication as well). As a university department that deals with graduate students, postdocs, visiting researchers, and various other sorts of ongoing or sporadic collaborations, we have a user population that's essentially impossible to keep track of centrally (and sometimes at all). So we want to be able to tell people things like 'this account that you sponsor doesn't seem to have been used for a year'.
As far as I can tell from Internet searches and so on, there are an assorted bunch of log aggregation, analysis, and querying tools. Logstash is the big one that many people have heard of, but then there's Graylog and fluentd and no doubt others. In theory any of these ought to be the solution to our issue. In practice, there seem to be two main drawbacks:
- They all seem to be designed for large to very large environments.
We have what I tend to call a midsized environment;
what's relevant here is that we only have on the order of 20 to
30 servers. Systems designed for large environments seem to be
both complicated and heavyweight, requiring things like JVMs and
multiple servers and so on.
- None of them appear to come with or have a comprehensive set of parsers to turn syslog messages from various common programs into the sort of structured information that these systems seem designed to work with. You can write your own parsers (usually with regular expressions), but doing that well requires a relatively deep knowledge of just what messages the programs can produce.
(In general all of these systems feel as if they're primarily focused on application level logging of structured information, where you have your website or backend processing system or whatever emit structured messages into the logging backend. Or perhaps I don't understand how you're supposed to use these systems.)
We can undoubtedly make these systems solve our problem. We can set
up the required collection of servers and services and get them
talking to each other (and our central syslog server),
and we can write a bunch of grok patterns
to crack apart
sshd and Dovecot and Samba messages. But all of
this feels as if we are using the back of a large and very sharp
axe to hammer in a small nail. It works, awkwardly, but it's probably
not the right way.
It certainly feels as if structured capturing and analysis of syslog
messages from common programs like
sshd, Dovecot, and so on in a
moderate sized environment ought to be a well solved problem. We
can't be the first people to want to do this, so this particular
wheel must have been reinvented repeatedly by now. But I can't find
even a collection of syslog parsing patterns for common Unix daemons,
much less a full system for this.
(If people know of systems or resources for doing this, we would of course be quite interested. There are some SaaS services that do log analysis for you, but as a university department we're not in a position to pay for this (staff time is free, as always).)