An rsyslog(d) syslog forwarding setup for Grafana Loki (via Promtail)
Suppose, not hypothetically, that you have a shiny new Grafana Loki setup to store and query your logs (or at least the logs that come from the systemd journal on your Linux machines, and maybe some additional log files on them). Also suppose that you have some OpenBSD machines whose logs you would like to get into Loki. OpenBSD doesn't have the systemd journal, or for that matter a build of Promtail, the Loki log-shipping client. However, Promtail can receive logs via syslog, and OpenBSD can send syslog logs to remote servers (which we're already using for our central syslog server). Unfortunately Promtail only accepts syslog messages in RFC 5424 format, and OpenBSD doesn't send that. Instead, OpenBSD syslog sends what is usually called RFC 3164 format, which is really "the BSD syslog protocol" written down for informational purposes. In order to send OpenBSD syslog to Promtail, we need a converter in the middle (which is the recommended configuration anyway).
In an ideal world there would be a simple standalone converter that did this for you. As far as I can tell, such a thing doesn't exist; instead, people doing this use a full scale syslog daemon with a minimal configuration file, generally either syslog-ng or rsyslog. Syslog-ng is more popular and you can find suitable configuration examples with Internet searches (eg); however, Ubuntu uses rsyslog by default and you can't install syslog-ng without throwing rsyslog out. Thus, rsyslog it is.
On Ubuntu 22.04 LTS, a minimal 'UDP or TCP syslog to RFC 5424 TCP syslog' forwarding configuration file is the following:
# Listen on UDP and TCP 50514 # forward to TCP 127.0.0.1:40514 module(load="imudp") module(load="imtcp") # Pick your port to taste input(type="imudp" port="50514") input(type="imtcp" port="50514") # Forward everything *.* action(type="omfwd" protocol="tcp" target="127.0.0.1" port="40514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on" action.resumeRetryCount="-1" queue.type="linkedlist" queue.size="50000")
Much of the
action(...) portion comes from the Promtail
output configuration for rsyslog.
I've added the queueing options to insure that we don't drop messages
if the local receiving Promtail is down for a bit (because it's being
restarted, or because we're starting during system boot).
In Promtail's configuration file, we need a syslog listener in the
scrape configurations, as documented in the
see also the syslog receiver
section. What I've set up is:
- job_name: syslog-receiver syslog: listen_address: 127.0.0.1:40514 # Don't disconnect the forwarder idle_timeout: 12h use_incoming_timestamp: true labels: job: syslog-receiver # Copy syslog bits to standard labels relabel_configs: - source_labels: ['__syslog_message_hostname'] target_label: host - source_labels: ['__syslog_message_severity'] target_label: level - source_labels: ['__syslog_message_facility'] target_label: syslog_facility - source_labels: ['__syslog_message_app_name'] target_label: syslog_identifier
There are some additional RFC 5424 syslog message fields that
Promtail can theoretically record, such as the procid (which is the
traditional syslog PID). In practice, you can't record the
because it will lead to a Loki cardinality explosion and syslog messages from OpenBSD
machines don't seem to have a msgid. OpenBSD
syslog messages also have no RFC 5424 structured data, so setting
label_structured_data' is pointless. You may also want to do
some label remapping on the reported host name, if (for example)
your OpenBSD machines or other syslog sources report their fully
qualified domain names while you normally use short names in your
Since our intermediate rsyslog may hold on to messages in case of problems, we set use_incoming_timestamp so that the syslog log messages carry their original incoming times instead of whenever our Promtail actually received them.
You can run the rsyslog forwarder and the Promtail it forwards to on more than one host if you want to. This may be useful if you have a Linux machine that has an interface on an otherwise isolated network segment with OpenBSD machines, which happens to describe one of our 'sandbox' networks. As far as Promtail goes, you can run a separate Promtail instance with just the syslog scrape configuration, or include the syslog scrape configuration in the host's standard Promtail setup. The latter is what we're going to do, since it involves fewer daemons and configuration files.
(This elaborates on this tweet of mine and then this one.)
Sidebar: Whether or not to label the Promtail host
If you're going to run multiple copies of this setup, you might or
might not want to include a label to identify which host and Promtail
instance logs were submitted through (for example, as a '
label). On the one hand, this lets you identify where any surprise
logs are coming from. On the other hand, this increases Loki label
cardinality if your syslog machines can naturally change which host
they forward their logs through.
In our environment we're probably going to add a '
because anything that's using a non-standard forwarding host will
be doing it because they can't reach the standard one (which is
also the Loki host).