Suppose, not hypothetically, that you have a shiny new Grafana Loki setup to store and query your logs (or at least the logs that come from the systemd journal on your Linux machines, and maybe some additional log files on them). Also suppose that you have some OpenBSD machines whose logs you would like to get into Loki. OpenBSD doesn't have the systemd journal, or for that matter a build of Promtail, the Loki log-shipping client. However, Promtail can receive logs via syslog, and OpenBSD can send syslog logs to remote servers (which we're already using for our central syslog server). Unfortunately Promtail only accepts syslog messages in RFC 5424 format, and OpenBSD doesn't send that. Instead, OpenBSD syslog sends what is usually called RFC 3164 format, which is really "the BSD syslog protocol" written down for informational purposes. In order to send OpenBSD syslog to Promtail, we need a converter in the middle (which is the recommended configuration anyway).

In an ideal world there would be a simple standalone converter that did this for you. As far as I can tell, such a thing doesn't exist; instead, people doing this use a full scale syslog daemon with a minimal configuration file, generally either syslog-ng or rsyslog. Syslog-ng is more popular and you can find suitable configuration examples with Internet searches (eg); however, Ubuntu uses rsyslog by default and you can't install syslog-ng without throwing rsyslog out. Thus, rsyslog it is.

On Ubuntu 22.04 LTS, a minimal 'UDP or TCP syslog to RFC 5424 TCP syslog' forwarding configuration file is the following:

# Listen on UDP and TCP 50514
# forward to TCP 127.0.0.1:40514

module(load="imudp")
module(load="imtcp")
# Pick your port to taste
input(type="imudp" port="50514")
input(type="imtcp" port="50514")

# Forward everything
*.*  action(type="omfwd"
       protocol="tcp" target="127.0.0.1" port="40514"
       Template="RSYSLOG_SyslogProtocol23Format"
       TCP_Framing="octet-counted" KeepAlive="on"
       action.resumeRetryCount="-1"
       queue.type="linkedlist" queue.size="50000")

Much of the action(...) portion comes from the Promtail output configuration for rsyslog. I've added the queueing options to insure that we don't drop messages if the local receiving Promtail is down for a bit (because it's being restarted, or because we're starting during system boot).

In Promtail's configuration file, we need a syslog listener in the scrape configurations, as documented in the syslog section; see also the syslog receiver section. What I've set up is:

- job_name: syslog-receiver
  syslog:
    listen_address: 127.0.0.1:40514
    # Don't disconnect the forwarder
    idle_timeout: 12h
    use_incoming_timestamp: true
    labels:
      job: syslog-receiver

  # Copy syslog bits to standard labels
  relabel_configs:
    - source_labels: ['__syslog_message_hostname']
      target_label: host
    - source_labels: ['__syslog_message_severity']
      target_label: level
    - source_labels: ['__syslog_message_facility']
      target_label: syslog_facility
    - source_labels: ['__syslog_message_app_name']
      target_label: syslog_identifier

There are some additional RFC 5424 syslog message fields that Promtail can theoretically record, such as the procid (which is the traditional syslog PID). In practice, you can't record the procid because it will lead to a Loki cardinality explosion and syslog messages from OpenBSD machines don't seem to have a msgid. OpenBSD syslog messages also have no RFC 5424 structured data, so setting 'label_structured_data' is pointless. You may also want to do some label remapping on the reported host name, if (for example) your OpenBSD machines or other syslog sources report their fully qualified domain names while you normally use short names in your host label.

Since our intermediate rsyslog may hold on to messages in case of problems, we set use_incoming_timestamp so that the syslog log messages carry their original incoming times instead of whenever our Promtail actually received them.

You can run the rsyslog forwarder and the Promtail it forwards to on more than one host if you want to. This may be useful if you have a Linux machine that has an interface on an otherwise isolated network segment with OpenBSD machines, which happens to describe one of our 'sandbox' networks. As far as Promtail goes, you can run a separate Promtail instance with just the syslog scrape configuration, or include the syslog scrape configuration in the host's standard Promtail setup. The latter is what we're going to do, since it involves fewer daemons and configuration files.

(This elaborates on this tweet of mine and then this one.)

Sidebar: Whether or not to label the Promtail host

If you're going to run multiple copies of this setup, you might or might not want to include a label to identify which host and Promtail instance logs were submitted through (for example, as a 'jobhost' label). On the one hand, this lets you identify where any surprise logs are coming from. On the other hand, this increases Loki label cardinality if your syslog machines can naturally change which host they forward their logs through.

In our environment we're probably going to add a 'jobhost' label because anything that's using a non-standard forwarding host will be doing it because they can't reach the standard one (which is also the Loki host).