If your environment is on the low end of the organized 'DevOps' spectrum, here's an easy improvement that I urge you to make right away: create a central syslog server that all of your machines echo their syslogs to. Disk space is cheap today and if you have an ordinary sized environment you don't need a particularly powerful machine or a particularly fast disk system (at least if you turn off synchronous syslog writes on your log host). Once you've set up the syslog server all you need to do to ship logs off to it is to add the following to everyone's syslog configuration:

*.*   @CENTRAL.SERVER.IP

(I prefer an IP address because it still works if hostname lookups blow up for some reason.)

Modern Linux versions of syslog support putting configuration fragments in a directory (typically /etc/rsyslog.d), so you can just make one and drop it in there; you don't even need to change the system-supplied syslog configuration files.

The syslog server itself should start with a simple configuration. First, log all of each facility's messages to a separate file (you might put all localN facilities in one file, depending). Then also log all messages to a single file; we call it allmessages. Your syslog server should have lots of disk space so you should configure syslog log rotation to keep many days or weeks of logs (trust me, this will be useful). Don't try to be clever by dropping debug messages or sorting things by priority and stuff like that. It's not worth the extra hassle and it's not reliable (plenty of programs log things you want to know at debug priority, or at random other priorities).

(You might wonder about separating things by facilities, but programs are much more consistent about that and it's useful to have a lower volume place to look for things like kernel messages.)

The important thing that this does is that it gives you a single place to get a global view on what happened when (at a system level) and to monitor your systems live if you need to. Over and over again it's been extremely helpful to us to see a coherent cross-system view and timeline of things like kernel error messages from NFS clients and NFS servers, authentication logs from multiple systems, and so on. The allmessages log itself gives us an easy way of seeing all syslog messages from a single machine no matter what random facility they were logged on, or all messages related to a single machine (both from the machine itself and from others interacting with it). Finally, you don't have to try to remember (or find, or fix) the syslog configuration on any particular machine and where it puts what messages; since all messages are shipped off to your central machine you can just look there and be done with it.

Of course this is not perfect. Sending syslog messages to a remote server is not completely reliable in general, things can go wrong with the network while the local disk is okay (although the converse can happen too), and so on and so forth. You can probably do better if you try hard enough. But if you don't have any sort of central logging right now, the perfect is the enemy of the good. Put together a central syslog machine now and improve it later. The benefit of having one right now is worth doing it twice (and worth it being less than perfect).

PS: If you're running enough Unix machines with enough syslog message volume that a central syslog server needs to be a beefy server with ultra-fast networking and many terabytes of disk space and so on, you hopefully already have some sort of central logging solution. If not, all I can suggest is multiple syslog aggregators to at least cluster things together.

PPS: Yes, there are some security concerns about a central syslog server. My personal view is that the security worries are worth the benefit of having a high quality central view.