Centralizing syslogs as an easy way to improve your environment
If your environment is on the low end of the organized 'DevOps' spectrum, here's an easy improvement that I urge you to make right away: create a central syslog server that all of your machines echo their syslogs to. Disk space is cheap today and if you have an ordinary sized environment you don't need a particularly powerful machine or a particularly fast disk system (at least if you turn off synchronous syslog writes on your log host). Once you've set up the syslog server all you need to do to ship logs off to it is to add the following to everyone's syslog configuration:
(I prefer an IP address because it still works if hostname lookups blow up for some reason.)
Modern Linux versions of syslog support putting configuration fragments
in a directory (typically
/etc/rsyslog.d), so you can just make one
and drop it in there; you don't even need to change the system-supplied
syslog configuration files.
The syslog server itself should start with a simple configuration.
First, log all of each facility's messages to a separate file (you might
localN facilities in one file, depending). Then also log
all messages to a single file; we call it
allmessages. Your syslog
server should have lots of disk space so you should configure syslog
log rotation to keep many days or weeks of logs (trust me, this will be
useful). Don't try to be clever by dropping
debug messages or sorting
things by priority and stuff like that. It's not worth the extra hassle
and it's not reliable (plenty of programs log things you want to know at
debug priority, or at random other priorities).
(You might wonder about separating things by facilities, but programs are much more consistent about that and it's useful to have a lower volume place to look for things like kernel messages.)
The important thing that this does is that it gives you a single
place to get a global view on what happened when (at a system level)
and to monitor your systems live if you need to. Over and over again
it's been extremely helpful to us to see a coherent cross-system
view and timeline of things like kernel error messages from NFS
clients and NFS servers, authentication logs from multiple systems,
and so on. The
allmessages log itself gives us an easy way of
seeing all syslog messages from a single machine no matter what
random facility they were logged on, or all messages related to a
single machine (both from the machine itself and from others
interacting with it). Finally, you don't have to try to remember
(or find, or fix) the syslog configuration on any particular machine
and where it puts what messages; since all messages are shipped off
to your central machine you can just look there and be done with
Of course this is not perfect. Sending syslog messages to a remote server is not completely reliable in general, things can go wrong with the network while the local disk is okay (although the converse can happen too), and so on and so forth. You can probably do better if you try hard enough. But if you don't have any sort of central logging right now, the perfect is the enemy of the good. Put together a central syslog machine now and improve it later. The benefit of having one right now is worth doing it twice (and worth it being less than perfect).
PS: If you're running enough Unix machines with enough syslog message volume that a central syslog server needs to be a beefy server with ultra-fast networking and many terabytes of disk space and so on, you hopefully already have some sort of central logging solution. If not, all I can suggest is multiple syslog aggregators to at least cluster things together.
PPS: Yes, there are some security concerns about a central syslog server. My personal view is that the security worries are worth the benefit of having a high quality central view.