perl: slice of a hash vs hash slice

perl hash slices are a way to access several elements of a hash simultaneously using a list of subscripts. It's more convenient than writing out the individual elements as a list of separate scalar values.

I recently needed to create a slice of a hash rather than a hash slice, ie. I wanted to create a hash with a subset of the key/value pairs from another hash.

Here's how I did it:

my %orginal = (foo => 'something', bar => 42, baz => 'king');
my %subset = map {$_ => $original{$_}} qw(foo bar));
# %subset is now: ( foo => 'something',  bar => 42 )

MCollective and other languages

I often get asked about MCollective and other programming languages. Thus far we only support Ruby but my hope is in time we’ll be able to be more generic.

Initially I had a few requirements from serialization:

  • It must retain data types
  • Encoding the same data – like a hash – twice should give the same result from the point of view of md5()

That was about it really. This was while we used a pre-shared key to validate requests and so the result of the encode and decode should be the same on the sender as on the receiver. With YAML this was never the case so I used Marshal.

We recently had a SSL based security plugin contributed that relaxed the 2nd requirement so we can go back to using YAML. We could in theory relax the 1st requirement but it would just inhibit the kind of tools you can build with MCollective quite a bit. So I’d strongly suggest this is a must have.

Today there are very few cross language serializers that let you just deal with arbitrary data YAML is one that seems to have a log of language support. Prior to version 1.0.0 of MCollective the SSL security system only supported Marshal but we’ll support YAML in addition to Marshal in 1.0.0.

This enabled me to write a Perl client that speaks to your standard Ruby collective (if it runs this new plugin).

You can see the Perl client here. The Perl code is roughly a mc-find-hosts written in Perl and without command line options for filtering – though you can just adjust the filters in the code. It’s been years since I wrote any Perl so that’s just the first thing that worked for me.

Point is someone should be able to take any language that has the Syck YAML bindings and write a client library to talk with Mcollective. I tried the non Syck bindings in PHP and it’s unusable, I suspect the PHP Syck bindings will work better but I didn’t try them.

As mentioned on the user list post 1.0.0 I intend to focus on long running and scheduled requests I’ll then also work on some kind of interface between Mcollective and Agents written in other languages – since that is more or less how long running scheduled tasks would work anyway. This will then use the Ruby as a transport hooking clients and agents in different languages together.

I can see that I’ll enable this but I am very unlikely to write the clients myself. I am therefore keen to speak to community members who want to speak to MCollective from languages like Python and who have some time to work on this.

Count of denied connections with iptables

In my iptables configurations, I generally allow all traffic I am interested in and deny the rest, logging anything that is denied.

I found that this can get a bit noisy with loads of connections to udp:137 and udp:500, etc. so I decided to deny the more common ports without logging. But which are the most common ports?

A typical log line looks like this:

Jan 12 09:42:01 b003 kernel: FAILSAFE -- DENY IN=bond0 OUT= MAC=00:26:b9:3d:fb:18:00:d0:c0:52:88:00:08:00 SRC=115.177.32.129 DST=a.b.c.d LEN=40 TOS=0x00 PREC=0x00 TTL=46 ID=22118 PROTO=TCP SPT=6643 DPT=20480 WINDOW=0 RES=0x00 ACK RST URGP=0

I wrote this perl one-liner to count no. of packets blocked per port:

perl -e 'while(<>) {if ($_ =~ /PROTO=([^ ]+).*DPT=([^ ]+)/) {$proto=$1 ; $port=$2; $count{"$proto:$port"}++}} foreach $key (sort { $count{$b} <=> $count{$a} } keys %count) {print "$key: $count{$key}\n"}' /var/log/messages | head -10

Sample output:

UDP:137: 226619
UDP:500: 107056
UDP:33436: 25244
UDP:33435: 22203
UDP:33437: 16035
TCP:20480: 10782
TCP:30080: 8291
UDP:33438: 8162
UDP:33439: 7268
UDP:33440: 5885

mysql-snmp 1.0 – SNMP monitoring for MySQL

I’m really proud to announce the release of the version 1.0 of mysql-snmp.

What is mysql-snmp?

mysql-snmp is a mix between the excellent MySQL Cacti Templates and a Net-SNMP agent. The idea is that combining the power of the MySQL Cacti Templates and any SNMP based monitoring would unleash a powerful mysql monitoring system. Of course this project favorite monitoring system is OpenNMS.

mysql-snmp is shipped with the necessary OpenNMS configuration files, but any other SNMP monitoring software can be used (provided you configure it).

To get there, you need to run a SNMP agent on each MySQL server, along with mysql-snmp. Then OpenNMS (or any SNMP monitoring software) will contact it and fetch the various values.

Mysql-snmp exposes a lot of useful values including but not limited to:

  • SHOW STATUS values
  • SHOW ENGINE INNODB STATUS parsed values (MySQL 5.0, 5.1, XtraDB or Innodb plugin are supported)

Here are some graph examples produced with OpenNMS 1.6.5 and mysql-snmp 1.0 on one of Days of Wonder MySQL server (running a MySQL 5.0 Percona build):

commands

This graph shows the number of SQL commands per unit of time

mem

You can see the effect of MySQL bug #47991

tmpinnodbwrites

graphtablelocks

Where to get it

mysql-snmp is available in my github repository. The repository contains a spec file to build a RPM and what is needed to build a Debian package. Refer to the README or the mysql-snmp page for more information.

Thanks to gihub, it is possible to download the tarball instead of using Git:

Mysql-snmp v1.0 tarball

Changelog

This lists all new features/options from the initial version v0.6:

  • Spec file to build RPM
  • Use of configuration file for storing mysql password
  • Fix of slave handling
  • Fix for mk-heartbeat slave lag
  • Support of InnoDB plugin and Percona XtraDB
  • Automated testing of InnoDB parsing
  • Removed some false positive errors
  • OpenNMS configuration generation from MySQL Cacti Templates core files
  • 64 bits computation done in Perl instead of (ab)using MySQL
  • More InnoDB values (memory, locked tables, …)

Reporting Issues

Please use Github issue system to report any issues.

Requirements

There is a little issue here. mysql-snmp uses Net-Snmp. Not all versions of Net-Snmp are supported as some older versions have some bug for dealing with Counter64. Version 5.4.2.1 with this patch is known to work fine.

Also note that this project uses some Counter64, so make sure you configure your SNMP monitoring software to use SNMP v2c or v3 (SNMP v1 doesn’t support 64 bits values).

Final words!

I wish everybody an happy new year. Consider this new version as my Christmas present to the community :-)

Masterzen’s Blog 2010-01-10 17:14:06

I’m really proud to announce the release of the version 1.0 of mysql-snmp.

What is mysql-snmp?

mysql-snmp is a mix between the excellent MySQL Cacti Templates and a Net-SNMP agent. The idea is that combining the power of the MySQL Cacti Templates and any SNMP based monitoring would unleash a powerful mysql monitoring system. Of course this project favorite monitoring system is OpenNMS.

mysql-snmp is shipped with the necessary OpenNMS configuration files, but any other SNMP monitoring software can be used (provided you configure it).

To get there, you need to run a SNMP agent on each MySQL server, along with mysql-snmp. Then OpenNMS (or any SNMP monitoring software) will contact it and fetch the various values.

Mysql-snmp exposes a lot of useful values including but not limited to:

  • SHOW STATUS values
  • SHOW ENGINE INNODB STATUS parsed values (MySQL 5.0, 5.1, XtraDB or Innodb plugin are supported)

Here are some graph examples produced with OpenNMS 1.6.5 and mysql-snmp 1.0 on one of Days of Wonder MySQL server (running a MySQL 5.0 Percona build):

commands

mem

tmp

innodbwrites

graph

tablelocks

Where to get it

mysql-snmp is available in my github repository. The repository contains a spec file to build a RPM and what is needed to build a Debian package. Refer to the README or the mysql-snmp page for more information.

Thanks to gihub, it is possible to download the tarball instead of using Git:

Mysql-snmp v1.0 tarball

Changelog

This lists all new features/options from the initial version v0.6:

  • Spec file to build RPM
  • Use of configuration file for storing mysql password
  • Fix of slave handling
  • Fix for mk-heartbeat slave lag
  • Support of InnoDB plugin and Percona XtraDB
  • Automated testing of InnoDB parsing
  • Removed some false positive errors
  • OpenNMS configuration generation from MySQL Cacti Templates core files
  • 64 bits computation done in Perl instead of (ab)using MySQL
  • More InnoDB values (memory, locked tables, …)

Reporting Issues

Please use Github issue system to report any issues.

Requirements

There is a little issue here. mysql-snmp uses Net-Snmp. Not all versions of Net-Snmp are supported as some older versions have some bug for dealing with Counter64. Version 5.4.2.1 with this patch is known to work fine.

Also note that this project uses some Counter64, so make sure you configure your SNMP monitoring software to use SNMP v2c or v3 (SNMP v1 doesn’t support 64 bits values).

Final words!

I wish everybody an happy new year. Consider this new version as my Christmas present to the community :–)