Custom Thresholding for specific nodes in OpenNMS

OpenNMS ships with the thresholds for some events already defined. For example, there is a memory threshold defined as:

<group name="netsnmp-memory-nonlinux" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" expression="memAvailReal / memTotalReal * 100.0" ds-type="node" ds-label="" value="5.0" rearm="10.0" trigger="2"/>
</group>

ie. if free memory drops below 5% then an event will be created. The alert will be cancelled automatically if free memory subsequently rises above 10%

I wanted to configure some specific nodes with a different threshold, eg. generate an event when free memory drops below 2.5%.

Here's what I did.

Add new Surveillance Category

Add nodes to new Surveillance Category

Add a new group to thresholds.xml:

    <group name="netsnmp-memory-linux-2.5" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" ds-type="node" value="2.5" rearm="5.0"
            trigger="2" filterOperator="or" expression="(memAvailReal + memCached) / memTotalReal * 100.0"/>
    </group>

Update threshd-configuration.xml to modify the existing netsnmp-memory-linux package and add a new netsnmp-memory-linux-2.5 package:

    <package name="netsnmp-memory-linux">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; ! ( catincNetSNMP-Mem-2_5 )</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux"/>
        </service>
    </package>

    <package name="netsnmp-memory-linux-2.5">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; catincNetSNMP-Mem-2_5</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux-2.5"/>
        </service>
    </package>

The tricky bit is the "catincNetSNMP-Mem-2_5". This is a function "catinc" which matches all nodes in the specified category ie. "NetSNMP-Mem-2_5" in this example. The first use of it is in the netsnmp-memory-linux category to exclude nodes in the NetSNMP-Mem-2_5 category. The second use is to include nodes in the NetSNMP-Mem-2_5 category.

mysql-snmp 1.0 – SNMP monitoring for MySQL

I’m really proud to announce the release of the version 1.0 of mysql-snmp.

What is mysql-snmp?

mysql-snmp is a mix between the excellent MySQL Cacti Templates and a Net-SNMP agent. The idea is that combining the power of the MySQL Cacti Templates and any SNMP based monitoring would unleash a powerful mysql monitoring system. Of course this project favorite monitoring system is OpenNMS.

mysql-snmp is shipped with the necessary OpenNMS configuration files, but any other SNMP monitoring software can be used (provided you configure it).

To get there, you need to run a SNMP agent on each MySQL server, along with mysql-snmp. Then OpenNMS (or any SNMP monitoring software) will contact it and fetch the various values.

Mysql-snmp exposes a lot of useful values including but not limited to:

  • SHOW STATUS values
  • SHOW ENGINE INNODB STATUS parsed values (MySQL 5.0, 5.1, XtraDB or Innodb plugin are supported)

Here are some graph examples produced with OpenNMS 1.6.5 and mysql-snmp 1.0 on one of Days of Wonder MySQL server (running a MySQL 5.0 Percona build):

commands

This graph shows the number of SQL commands per unit of time

mem

You can see the effect of MySQL bug #47991

tmpinnodbwrites

graphtablelocks

Where to get it

mysql-snmp is available in my github repository. The repository contains a spec file to build a RPM and what is needed to build a Debian package. Refer to the README or the mysql-snmp page for more information.

Thanks to gihub, it is possible to download the tarball instead of using Git:

Mysql-snmp v1.0 tarball

Changelog

This lists all new features/options from the initial version v0.6:

  • Spec file to build RPM
  • Use of configuration file for storing mysql password
  • Fix of slave handling
  • Fix for mk-heartbeat slave lag
  • Support of InnoDB plugin and Percona XtraDB
  • Automated testing of InnoDB parsing
  • Removed some false positive errors
  • OpenNMS configuration generation from MySQL Cacti Templates core files
  • 64 bits computation done in Perl instead of (ab)using MySQL
  • More InnoDB values (memory, locked tables, …)

Reporting Issues

Please use Github issue system to report any issues.

Requirements

There is a little issue here. mysql-snmp uses Net-Snmp. Not all versions of Net-Snmp are supported as some older versions have some bug for dealing with Counter64. Version 5.4.2.1 with this patch is known to work fine.

Also note that this project uses some Counter64, so make sure you configure your SNMP monitoring software to use SNMP v2c or v3 (SNMP v1 doesn’t support 64 bits values).

Final words!

I wish everybody an happy new year. Consider this new version as my Christmas present to the community :-)