Chris's Wiki :: blog/sysadmin/ModestScaleSyslogAnalysis Commentshttps://utcc.utoronto.ca/~cks/space/blog/sysadmin/ModestScaleSyslogAnalysis?atomcommentsDWiki2016-10-05T16:40:24ZRecent comments in Chris's Wiki :: blog/sysadmin/ModestScaleSyslogAnalysis.By liam at unc edu on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:5f680ad5b3e31ffe4da4b24d7f5ab61e4b746c60liam at unc edu<div class="wikitext"><p>Splunk is an option for you if you log less than 500MB a day.
<a href="https://www.splunk.com/en_us/products/splunk-enterprise/free-vs-enterprise.html">https://www.splunk.com/en_us/products/splunk-enterprise/free-vs-enterprise.html</a></p>
<p>The free-as-in-beer tier is probably too low volume for you - but it's a good tool if you have small enough needs (it's a good tool if you have large needs and a budget too :-))</p>
</div>2016-10-05T16:40:24ZBy erlogan on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:820adcfbc8bfc3f12a1fc97d73421fd5693f6035erlogan<div class="wikitext"><p>The Elasticsearch/Logstash/Kibana ("ELK") suite is popular in industry for this purpose. All the various centralized pieces can be set to run on the same machine, and I have done so on relatively modest hardware for clients in the past. It's quite good at either running predefined filters or handcrafted ones against source data to extract salient fields.</p>
<p>It does require a JVM, but I find this less onerous than I used to.</p>
</div>2016-09-30T18:32:01ZBy edgar n. on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:fe074803f50e2b6fbb06ad517285330ac7b57f1aedgar n.<div class="wikitext"><p>Alternatively, perhaps use some mechanism (perhaps like that pam lastlog thing posted in the first comment?) to just track this on each system. Could use that lastlog mechanism or use a log monitor like SEC to make a new log file of people's logins on each system. I know SEC can be setup to only log for a given thing once in a specified time period. </p>
<p>Hmm, perhaps use SEC to feed logstash the exact info you want so don't have to learn too much about logstash formats?</p>
</div>2016-09-30T16:13:41ZBy Chris Siebenmann on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:28e01c104e6bdfb633e1263d3ced24fc63db1742Chris Siebenmann<div class="wikitext"><p>Unfortunately lastlog isn't suitable for us for a number of reasons,
including that it's not centralized and that it doesn't capture all
'login' events (eg some ssh connections won't create lastlog entries).</p>
<p>dozzie: thank you, that's all useful information and logdevourer looks
interesting. It looks like we'd still need to write patterns for all of
the syslog stuff we're interested in, though, unless there's a collection
of them somewhere.</p>
<p>(Our approach for ingestion will probably be to have all of the machines
forward their syslog logs to the ingestion point (in addition to our
central log server), since this seems relatively easy. We could also
process the central syslog logs once a day when they get rolled over,
but direct transmission seems more fire-and-forget.)</p>
</div>2016-09-30T16:08:19ZBy dozzie on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:b4111233b87bc40ac361ef7effbfa94890c39158dozzie<div class="wikitext"><p>First of all, Fluentd and logstash are message routers, where message is
JSON-compatibile data. These routers don't collect (normally shouldn't
collect) logs at all. Their purpose is to pass whatever message you submit to
wherever it should go (Fluentd uses categories, which are external to the
message itself).</p>
<p>This means that (a) Fluentd and logstash can be used for whatever messages you
generate (logs, monitoring, hardware/software inventory, others) and (b) for
logs alone you can be perfectly happy with syslog's own mechanisms.</p>
<p>Now there's the issue of parsing unstructured logs to JSON-compatible data
structure. If you go with Fluentd/logstash, you probably need to parse the
logs before submitting them to FL/L, and for this I have written a daemon,
<a href="https://github.com/dozzie/logdevourer">logdevourer</a> (it uses
<a href="http://www.liblognorm.com/">liblognorm</a> instead of regexps, which is
significantly better in the longer use). If you go with syslog forwarding, you
can set up log parsing in a central place (if at all; about that a little
later).</p>
<p>Let's assume you have logs parsed already. This time you need to store them
somewhere. People usually use ElasticSearch for this (with Kibana as a web
interface), since it's a search engine and stores JSON documents. This is also
more or less what Graylog2 does (or at least used to do three years ago, when
I last checked). But since the logs are JSON documents, they can be stored in
any document store, like MongoDB, CouchDB, or PostgreSQL (json/jsonb columns),
or even in flat files.</p>
<p>If you went with plain syslog forwarding, it can still be put into
ElasticSearch with some basic parsing (e.g. date, host, program, message), and
rsyslog even has a plugin for sending logs to ES. Unfortunately I'm not sure
how much gain there will be (single ES instance used to be slower than
grepping through flat files, but it provides a query language, so I considered
that as a progress).</p>
<p>I'm not sure if I would deploy ElasticSearch for dozen servers (probably yes,
because query language and Kibana as a somewhat sensible web interface). For
Fluentd/logstash, I probably would use them, especially that I could use them
for monitoring system as well. But deploying a central syslog server is
certainly a low-hanging fruit.</p>
<p>I hope my explaination will be of some use to you.</p>
</div>2016-09-30T13:54:59ZBy Christian Neukirchen on /blog/sysadmin/ModestScaleSyslogAnalysistag:CSpace:blog/sysadmin/ModestScaleSyslogAnalysis:579c1c0f183af14c067e547899242ef06dccad6dChristian Neukirchenhttp://chneukirchen.org/<div class="wikitext"><p>Sounds like something that can be done with pam_lastlog? <a href="http://captainslog.me/2016/07/dovecot-update-lastlog/">http://captainslog.me/2016/07/dovecot-update-lastlog/</a></p>
</div>2016-09-30T11:06:35Z