Brief notes on making Prometheus's SNMP exporter use additional SNMP MIB(s)

September 29, 2024

Suppose, not entirely hypothetically, that you have a DSL modem that exposes information about the state of your DSL link through SNMP, and you would like to get that information into Prometheus so that you could track it over time (for reasons). You could scrape this information by 'hand' using scripts, but Prometheus has an officially supported SNMP exporter. Unfortunately, in practice the Prometheus SNMP exporter pretty much has a sign on the front door that says "no user serviceable parts, developer access only"; how you do things with it if its stock configuration doesn't meet your needs is what I would call rather underdocumented.

The first thing you'll need to do is find out what generally known and unknown SNMP attributes ('OIDs') your device exposes. You can do this using tools like snmpwalk, and see also some general information on reading things over SNMP. Once you've found out what OIDs your device supports, you need to find out if there are public MIBs for them. In my case, my DSL modem exposed information about network interfaces in the standard and widely available 'IF-MIB', and ADSL information in the standard but not widely available 'ADSL-LINE-MIB'. For the rest of this entry I''ll assume that you've managed to fetch the ADSL-LINE-MIB and everything it depends on and put them in a directory, /tmp/adsl-mibs.

The SNMP exporter effectively has two configuration files (as I wrote about recently); a compiled ('generated') configuration file (or set of them) that lists in exhausting detail all of the SNMP OIDs to be collected, and an input file to a separate tool, the generator, that creates the compiled main file. To collect information from a new MIB, you need to set up a new SNMP exporter 'module' for it, and specify the root OID or OIDs involved to walk. This looks like:

---
modules:
  # The ADSL-LINE-MIB MIB
  adsl_line_mib:
    walk:
      - 1.3.6.1.2.1.10.94
      # or:
      #- adslMIB

Here adsl_line_mib is the name of the new SNMP exporter module, and we give it the starting OID of the MIB. You can't specify the name of the MIB itself as the OID to walk, although this is how 'snmpwalk' will present it. Instead you have to use the MIB's 'MODULE-IDENTITY' line, such as 'adslMIB'. Alternately, perusal of your MIB and snmpwalk results may suggest alternate names to use, such as 'adslLineMib'. Using the top level OID is probably easier.

The name of your new module is arbitrary, but it's conventional to use the name of the MIB in this form. You can do other things in your module; reading the existing generator.yml is probably the most useful documentation. As various existing modules show, you can walk multiple OIDs in one module.

This configuration file leaves out the 'auths:' section from the main generator.yml, because we only need one of them, and what we're doing is generating an additional configuration file for snmp_exporter that we'll use along with the stock snmp.yml. To actually generate our new snmp-adsl.yml, we do:

cd snmp_exporter/generator
go build
make # builds ./mibs
./generator generate \
   -m ./mibs \
   -m /tmp/adsl-mibs \
   -g generator-adsl.yml
   -o /tmp/snmp-adsl.yml

We give the generator both its base set of MIBs, which will define various common things, and the directory with our ADSL-LINE-MIB and all of the MIBs it may depend on. Although the input is small, the snmp-adsl.yml will generally be quite big; in my case, over 2,000 lines.

As I mentioned the other day, you may find that some of the SNMP OIDs actually returned by your device don't conform to the SNMP MIB. When this happens, your scrape results will not be a success but instead a HTTP 500 error with text that says things like:

An error has occurred while serving metrics:
error collecting metric Desc{fqName: "snmp_error", help: "BITS type was not a BISTRING on the wire.", constLabels: {}, variableLabels: {}}: error for metric adslAturCurrStatus with labels [1]: <nil>

This says that the the actual OID(s) for adslAturCurrStatus from my actual device didn't match what the MIB claimed. In this case, my raw snmpwalk output for this OID is:

.1.3.6.1.2.1.10.94.1.1.3.1.6.1 = BITS: 00 00 00 01 31

(I don't understand what this means, since I'm not anywhere near an SNMP expert.)

If the information is sufficiently important, you'll need to figure out how to modify either the MIB or the generated snmp-adsl.yml to get the information without snmp_exporter errors. Doing so is far beyond the scope of this entry. If the information is not that important, the simple way is to exclude it with a generator override:

---
modules:
  adsl_line_mib:
    walk:
      # ADSL-LINE-MIB
      #- 1.3.6.1.2.1.10.94
      - adslMIB
    overrides:
     # My SmartRG SR505N produces values for this metric
     # that make the SNMP exporter unhappy.
     adslAturCurrStatus:
       ignore: true

You can at least get the attribute name you need to ignore from the SNMP exporter's error message. Unfortunately this error message is normally visible only in scrape output, and you'll only see it if you scrape manually with something like 'curl'.

Written on 29 September 2024.
« Options for adding IPv6 networking to your libvirt based virtual machines
Resetting the backoff restart delay for a systemd service »

Page tools: View Source.
Search:
Login: Password:

Last modified: Sun Sep 29 23:13:36 2024
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.