Wandering Thoughts archives

Brief notes on how the Prometheus SNMP exporter's configurations work

A variety of devices (including DSL modems) expose interesting information via SNMP (which is not simple, despite its name). If you have a Prometheus environment, it would be nice to get (some of) this information from your SNMP capable devices into Prometheus. You could do this by hand with scripts and commands like 'snmpget', but there is also the officially supported SNMP exporter. Unfortunately, in practice the Prometheus SNMP exporter pretty much has a sign on the front door that says "no user serviceable parts, developer access only". Understanding how to do things even a bit out of standard with it is, well, a bit tricky. So here are some notes.

The SNMP exporter ships with a 'snmp.yml' configuration file that's what the actual 'snmp_exporter' program uses at runtime (possibly augmented by additional files you provide). As you'll read when you look at the file, this file is machine generated. As far as I can tell, the primary purpose of this file is to tell the exporter what SNMP OIDs it could try to read from devices, what metrics generated from them should be called, and how to interpret the various sorts of values it gets back over SNMP (for instance, network interfaces have a 'ifType' that in raw format is a number, but where the various values correspond to different types of physical network types). These SNMP OIDs are grouped into 'modules', with each module roughly corresponding to a SNMP MIB (the correspondence isn't necessarily exact). When you ask the SNMP exporter to query a SNMP device, you normally tell the exporter what modules to use, which determines what OIDs will be retrieved and what metrics you'll get back.

The generated file is very verbose, which is why it's generated, and its format is pretty underdocumented, which certainly does help contribute to the "no user serviceable parts" feeling. There is very little support for directly writing a new snmp.yml module (which you can at least put in a separate 'snmp-me.yml' file) if you happen to have a few SNMP OIDs that you know directly, don't have a MIB for, and want to scrape and format specifically. Possibly the answer is to try to write a MIB yourself and generate a snmp-me.yml from it, but I haven't had to do this so I have no opinions on which way is better.

The generated file and its modules are created from various known MIBs by a separate program, the generator. The generator has its own configuration file to describe what modules to generate, what OIDs go into each module, and so on. This means that reading generator.yml is the best way to find out what MIBs the SNMP exporter already supports. As far as I know, although generator.yml doesn't necessarily specify OIDs by name, the generator requires MIBs for everything you want to be in the generated snmp.yml file and generate metrics for.

The generator program and its associated data isn't available as part of the pre-built binary SNMP exporter packages. If you need anything beyond the limited selection of MIBs that are compiled into the stock snmp.yml, you need to clone the repository, go to the 'generator' subdirectory, build the generator with 'go build' (currently), run 'make' to fetch and process the MIBs it expects, get (or write) MIBs for your additional metrics, and then write yourself a minimal generator-me.yml of your own to add one or more (new) modules for your new MIBs. You probably don't want to regenerate the main snmp.yml; you might as well build a 'snmp-me.yml' that just has your new modules in it, and run the SNMP exporter with snmp-me.yml as an additional configuration file.

As a practical matter, you may find that your SNMP capable device doesn't necessarily conform to the MIB that theoretically describes it, including OIDs with different data formats (or data) than expected. In the simple case, you can exclude OIDs or named attributes from being fetched so that the non-conformance doesn't cause the SNMP exporter to throw errors:

modules:
  adsl_line_mib:
[...]
    overrides:
     adslAturCurrStatus:
       ignore: true

More complex mis-matches between the MIB and your device will have you reading whatever you can find for the available options for generator.yml or even for snmp.yml itself. Or you can change your mind and scrape through scripts or programs in other languages instead of the SNMP exporter (it's what we do for some of our machine room temperature sensors).

(I guess another option is editing the MIB so that it corresponds to what your device returns, which should make the generator produce a snmp-me.yml that matches what the SNMP exporter sees from the device.)

PS: A peculiarity of the SNMP exporter is that the SNMP metrics it generates are all named after their SNMP MIB names, which produce metric names that are not at all like conventional Prometheus metric names. It's possible to put a common prefix, such as 'snmp_metric_', on all SNMP metrics to make them at least a little bit better. Technically this is a peculiarity of snmp.yml, but changing it is functionally impossible unless you hand-edit your own version.