OpenNMS ships with the thresholds for some events already defined. For example, there is a memory threshold defined as:

<group name="netsnmp-memory-nonlinux" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" expression="memAvailReal / memTotalReal * 100.0" ds-type="node" ds-label="" value="5.0" rearm="10.0" trigger="2"/>
</group>

ie. if free memory drops below 5% then an event will be created. The alert will be cancelled automatically if free memory subsequently rises above 10%

I wanted to configure some specific nodes with a different threshold, eg. generate an event when free memory drops below 2.5%.

Here's what I did.

Add new Surveillance Category

Add nodes to new Surveillance Category

Add a new group to thresholds.xml:

    <group name="netsnmp-memory-linux-2.5" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" ds-type="node" value="2.5" rearm="5.0"
            trigger="2" filterOperator="or" expression="(memAvailReal + memCached) / memTotalReal * 100.0"/>
    </group>

Update threshd-configuration.xml to modify the existing netsnmp-memory-linux package and add a new netsnmp-memory-linux-2.5 package:

    <package name="netsnmp-memory-linux">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; ! ( catincNetSNMP-Mem-2_5 )</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux"/>
        </service>
    </package>

    <package name="netsnmp-memory-linux-2.5">
        <filter>IPADDR != '0.0.0.0' &amp; nodeSysOID == '.1.3.6.1.4.1.8072.3.2.10' &amp; catincNetSNMP-Mem-2_5</filter>
        <include-range begin="1.1.1.1" end="254.254.254.254"/>
        <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff" />
        <service name="SNMP" interval="300000" user-defined="false" status="on">
            <parameter key="thresholding-group" value="netsnmp-memory-linux-2.5"/>
        </service>
    </package>

The tricky bit is the "catincNetSNMP-Mem-2_5". This is a function "catinc" which matches all nodes in the specified category ie. "NetSNMP-Mem-2_5" in this example. The first use of it is in the netsnmp-memory-linux category to exclude nodes in the NetSNMP-Mem-2_5 category. The second use is to include nodes in the NetSNMP-Mem-2_5 category.

Earlier this month, I completed a switch to DNSimple for my domain’s DNS provider. I am still happy with the switch, and finally, just now, got around to writing a recipe to have my systems automatically register themselves in DNS.

In the post, I described automatically adding the DNS entries with the dnsimple cookbook. I did this as a proof of concept, but I didn’t add it to all my nodes, instead using my existing data bag-driven solution.

That said, this post serves as a brief document on how you can mimic this behavior with your own environment.

Encrypted Data Bag

I put my DNSimple credentials in an encrypted data bag. Since I have to decrypt and read the entire thing anyway, I also store the relevant data there. I keep my encrypted data bags in a bag called secrets. The structure looks like this:

1
2
3
4
5
6
7

{
  "id": "dnsimple",
  "api_token": "DNSimple API Token Here",
  "domain": "your-domain.example.com",
  "username": "DNSimple username",
  "password": "DNSimple password"
}

Replace the values with your values. Encrypting the data is optional, but requires that you create a secret key or key file. Read my previous post on the topic for more information.

1
2

% knife data bag from file secrets dnsimple.json
% knife data bag from file secrets dnsimple.json –secret-file /etc/chef/encrypted_data_bag_secret

The first command just uploads the data bag item, the second encrypts it. Note that I manage my workstation with Chef, so I will use the same secret file as the Chef default. The secret file needs to be copied to each system that will need it.

Recipe

The recipe looks like this:

1
2
3
4
5
6
7
8
9

dnsimple = encrypted_data_bag_item("secrets", "dnsimple")
dnsimple_record "#{node['hostname']}.int.#{dnsimple['domain']}" do
  content node['ipaddress']
  type "A"
  action :create
  username dnsimple['username']
  password dnsimple['password']
  domain dnsimple['domain']
end

As mentioned in the previous blog post, the encrypted_data_bag_item method is in a library. Either add that library to your cookbook, or use the class directly.

1

dnsimple = Chef::EncryptedDataBagItem.load("secrets", "dnsimple")

If you’re not using an encrypted data bag, then the item can be accessed with the normal method:

1

dnsimple = data_bag_item("bagname", "dnsimple")

The real work happens in the dnsimple_record LWRP, which will add an “A” record for the system running the recipe. Note that the actual entry is going to use the int subdomain, and it will use the domain stored in the data bag item. It also will use the default IP address of the node, which means the IP for the interface with the default route.

A new “which tool is best” battle is raging in the internets amongst developers and system administrators. The contestants are screen and tmux, and the jury is still out. This is very much an argument over what color to paint the bikeshed, but with the latest version of iTerm2, I think tmux is even more compelling. Personally, I chose tmux awhile ago.

At my day job, I worked with a customer that uses tmux for remote pairing between developers. At the time, tmux had better customizability, and better split-pane support (screen didn’t yet have vertical split). I stuck with tmux ever since, and was very pleased when an iTerm2 update announced integration with tmux.

iTerm2

For those who aren’t aware, iTerm2 is an alternate terminal program for Mac OS X. It is actually an updated codebase from the original, iTerm, which is effectively unmaintained. iTerm2 offers a lot of excellent features like split panes, Growl support, and many more.

One of the excellent new features is integration with tmux.

iTerm2’s tmux integration

If you already have iTerm2 installed, you may have seen the update check prompt you to update. You also need to install a special version of tmux that has the integration patched. The iTerm2 author is working with the tmux author to get this into the latest tmux codebase, so hopefully the custom compiled version won’t be necessary soon.

Using the new feature is relatively straightforward. Start up iTerm2 like normal. Then run tmux -C to open a new iTerm2 window that works like tmux.

Use the tmux menu in iTerm2 to open new windows in tmux. Note that there are keyboard shortcuts for each of these, and they are not the same as the tmux window commands.

You can also attach to a tmux session running in iTerm2. In the screenshot, this is running on the same system, for example purposes. However, since OS X has SSH, this can be useful if you want to SSH to another system in the local network and connect to the running session. For example, the system shown below is my wife’s iMac over screensharing, but I wouldn’t need to use screenshare (or participate in its lag) to connect to this anymore. The same holds true for connecting to my work laptop if necessary.

In this final example screenshot, you can see that I have multiple panes split in one iTerm2 tab. These correspond to the split windows in the attached tmux in the other window. Also, the two tabs in the iTerm2 window are separate tmux windows (0:zsh and 1:zsh).

And now, I can SSH to that system and attach to the tmux session started by iTerm2.

Automating Installation with Chef

Installing OS X apps is quite easy, but I automate them with Chef anyway. While it is a simple “install update and restart”, with a couple commands to install the update, I do have three systems I want this on. I updated my iterm2 cookbook to support installing the tmux integration for iTerm2. This is disabled by default, so it needs to be enabled via a node attribute. For example, I have this in my workstation role applied to my OS X workstations.

1
2
3
4
5
6
7
8

name "workstation"
description "Mac OS X workstations"
run_list("recipe[tmux]")
default_attributes(
  "iterm2" => {
    "tmux_enabled" => true
  }
)

Check out the iterm2 cookbook’s README for more information.

Earlier this week I wrote about Oldskool which is a Gem extendable search tool. Today I want to show how to create a plugin for it to query some custom source.

We’ll build a plugin that shows Puppet Type references, you can see how it will look in the image, click for a larger version.

The end result is that I can just search for “type exec” to get the correct documentation for my Puppet install. I’ll go quite quick through all the various bits here, the complete working plugin is in my GitHub.

The nice thing about rendering the type references locally is that you can choose exactly which version to render the docs for and you could possibly also render docs for locally written types that are not part of Puppet – not tried to see how you might render custom types though.

Plugins are made up of a few things that should have predictable names, in the case of our Puppet plugin I decided to call it puppet which means we need a class called Oldskool::PuppetHandler that does the work for that plugin. You can see the one here and it goes in lib/oldskool/puppet_handler.rb in your gem:

module Oldskool
  class PuppetHandler
    def initialize(params, keyword, config)
      @params = params
      @keyword = keyword
      @config = config
      self
    end
 
    def plugin_template(template)
      File.read(File.expand_path("../../../views/#{template}.erb", __FILE__))
    end
 
    def handle_request(keyword, query)
      type = Puppetdoc.new(query)
 
      menu = [{:title => "Type Reference", :url => "http://docs.puppetlabs.com/references/stable/type.html"},
              {:title => "Function Reference", :url => "http://docs.puppetlabs.com/references/stable/function.html"},
              {:title => "Language Guide", :url => "http://docs.puppetlabs.com/guides/language_guide.html"}]
 
      {:template => plugin_template(:type), :type => type.doc, :topmenu => menu}
    end
  end
end

The initialize and plugin_template methods will rarely change, the handle_request is where the magic happens. It gets called with the keyword and the query, so I set this up to respond to searched like type exec. If you needed any kind of configuration data from the main Oldskool config file you’d just add data to that YAML file and the data would be available in @config.

The keyword would be type and the query would be exec. The idea is that we could route for example type as well as function keywords into the plugin and then do different things with the query string.

I wrote a class called Puppetdoc that takes care of the Puppet introspection, I won’t go into the details but you can see it here, it just returns a hash with all the Markdown for each parameter, meta parameter and the type itself.

We then create a simple menu that’s just an array of title and url pairs that will be used to build the top menu that you see in the screenshot.

And finally we just return a hash. The hash that you return must include a template key the rest is optional, I override the meaning of the word template a bit – probably should have chosen a better name:

• If it’s a string it’s assumed the string is a ERB template, Sinatra will just render that
• When it’s the symbol :redirect then your hash must have a :url item in it, this will just redirect the user to another url
• When it’s the symbol :error or just nil you can optionally add a :error key that will be shown to the user

You can see in the code above I passed the menu in as :topmenu you could also pass it back as :sidemenu which will create buttons down the side of the page, you can use both menus and buttons at the same time.

This takes care of creating the data to display but not yet the displaying aspect. The call to plugin_template(:type) will read the contents of the type.erb in the plugins view directory and return the contents. The Oldskool framework will then render that template making your returned hash available in @result

Here’s the first part of the view in question, you can see the whole thing here:

<% unless @error %>
  <h2><%= @result[:type][:name].to_s.capitalize %> version <%= @result[:type][:version] %></h2>
<% end %>

Your view can check if @error is set to show some text to the user in the case of exceptions etc otherwise just display the results. You can see here the @result variable is the data the handle_request returned.

Finally there’s a convention for gem names – this one would be oldskool-puppet so you should create a similarly named Ruby file to load the various bits, place this in lib/oldskool-puppet.rb:

require 'oldskool/puppetdoc'
require 'oldskool/puppet_handler'

From there you just need to build the gem, the Rakefile below does that:

require 'rubygems'
require 'rake/gempackagetask'
 
spec = Gem::Specification::new do |spec|
  spec.name = "oldskool-puppet"
  spec.version = "0.0.3"
  spec.platform = Gem::Platform::RUBY
  spec.summary = "oldskool-1assword"
  spec.description = "description: Generate documentation for Puppet types"
 
  spec.files = FileList["lib/**/*.rb", "views/*.erb"]
  spec.executables = []
 
  spec.require_path = "lib"
 
  spec.has_rdoc = false
  spec.test_files = nil
  spec.add_dependency 'puppet'
  spec.add_dependency 'redcarpet'
 
  spec.extensions.push(*[])
 
  spec.author = "R.I.Pienaar"
  spec.email = "rip@devco.net"
  spec.homepage = "http://devco.net/"
end
 
Rake::GemPackageTask.new(spec) do |pkg|
  pkg.need_zip = false
  pkg.need_tar = false
end

% rake gem
mkdir -p pkg
WARNING:  no rubyforge_project specified
  Successfully built RubyGem
  Name: oldskool-puppet
  Version: 0.0.3
  File: oldskool-puppet-0.0.3.gem
mv oldskool-puppet-0.0.3.gem pkg/oldskool-puppet-0.0.3.gem
% gem push pkg/oldskool-puppet-0.0.3.gem

If your gem command is setup this will publish the gem to Github ready for use. In this case all I did was add it to my Gemfile for my webapp:

gem 'puppet', '2.6.9'
gem 'facter'
gem 'oldskool-puppet', '>= 0.0.3'

And used bundler to update my site after that everything worked.

I recently acquired the Remote Management card for my HP Microserver, which allows remote KVM & power control, IPMI management and hardware monitoring through temperature & fan sensors.

Note the extra connector on the card in addition to the standard PCI-e x1 connector which matches the dedicated slot on the Microserver motherboard. This presented a bit of a problem as I was using the space for the battery backup module for the RAID controller in the neighbouring slot.

Thankfully the long ribbon cable meant I could route the battery up to the space behind the DVD burner freeing the slot again. Once the card was installed and everything screwed back together I booted straight back into CentOS. Given IPMI is touted as a feature I figured that was the first thing to try so I installed OpenIPMI:

# yum -y install OpenIPMI ipmitool
...
# service ipmi start
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Error sending Chassis Status command

Hmm, not good. Looking at dmesg shows the following is output when the IPMI drivers get loaded:

ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified smic state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi_si: Interface detection failed
ipmi_si: Trying ACPI-specified smic state machine at mem address 0x0, slave address 0x0, irq 0
Could not set up I/O space
ipmi device interface

From reading the PDF manual it states that the IPMI KCS interface is at 0xCA2 in memory, not 0xCA8 that the kernel is trying to probe. Looking at the output from dmidecode shows where this value is probably coming from:

# dmidecode --type 38
# dmidecode 2.11
SMBIOS 2.6 present.
 
Handle 0x001B, DMI type 38, 18 bytes
IPMI Device Information
	Interface Type: KCS (Keyboard Control Style)
	Specification Version: 1.5
	I2C Slave Address: 0x10
	NV Storage Device: Not Present
	Base Address: 0x0000000000000CA8 (I/O)
	Register Spacing: Successive Byte Boundaries

This suggests a minor bug in the BIOS. Querying the ipmi_si module with modinfo shows it can be persuaded to use a different I/O address so I created /etc/modprobe.d/ipmi.conf containing the following:

1

options ipmi_si type=kcs ports=0xca2

Then bounce the service to reload the modules and try again:

# service ipmi restart
Stopping all ipmi drivers:                                 [  OK  ]
Starting ipmi drivers:                                     [  OK  ]
# ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
# ipmitool sdr
Watchdog         | 0x00              | ok
CPU_THEMAL       | 32 degrees C      | ok
NB_THERMAL       | 35 degrees C      | ok
SEL Rate         | 0 messages        | ok
AMBIENT_THERMAL  | 20 degrees C      | ok
EvtLogDisabled   | 0x00              | ok
System Event     | 0x00              | ok
SYS_FAN          | 1000 RPM          | ok
CPU Thermtrip    | 0x00              | ok
Sys Pwr Monitor  | 0x00              | ok

Success! With that sorted, you can now use ipmitool to further configure the management card, although not all of the settings are accessible such as IPv6 network settings so you have to use the BIOS or web interface for some of it.

Overall, I’m fairly happy with the management card. It has decent IPv6 support and the Java KVM client works okay on OS X should I ever need it but I couldn’t coax the separate virtual media client to work, I guess only Windows is supported.

Tmux has been getting a lot of attention lately. As George Nachman works toward a deeper integration of iTerm2 with tmux, more people are becoming aware of the ‘other’ terminal multiplexer. Around the office, people have been asking how I use tmux. While I’m also an avid iTerm2 user, I’m not sure if the new ‘deep integration’ will be worth the additional complexity. Tmux is useful enough on its own.

As a professional problem solver, I spend a considerable portion of my day working at a command prompt. Lately, I have been doing this work within a terminal multiplexer called tmux. Here are some tips for getting started.

Installing tmux

I use homebrew to manage packages on my machine. Installing tmux is as simple as:
brew install tmux

Source packages can also be found on the project website.

Getting started with tmux

Concepts

Before diving into the details of using tmux, it’s good to have a mental model of how some of the pieces fit together.

Tmux operates on a client/server model. A tmux server hosts any number of sessions. A session is a group of one or more windows. A window may contain one or more panes. Unlike screen, tmux considers windows and panes to be different types of entities. A pane may be broken out of a window into a new window, and a window’s pane may be joined to another window, but windows and panes are distinct.

Multiple clients can connect to the same server and access either the same or different sessions (and switch between them). Clients connect to the server via a socket. File permissions on the socket limit access to the tmux server. Read write access is needed for an interactive session. You can also have multiple servers running on different sockets if you need.

Let’s review with an example and get started. If you run tmux for the first time without any arguments it will start a new server, a new session containing one window that contains one pane in which your default-shell is running, AND a new client connected to that session on that server over a socket (probably somewhere in /tmp).

Phew.

If you detach this client from the server without explicitly closing the session, the session (and the server) will continue running (below we’ll see how this can become a problem if you’re not careful).

OK, Let’s get started.

Defaults

Tmux’s default ‘prefix’ is Ctrl + b (C-b). This works fine for me, and is a welcome change from screen’s default prefix of Ctrl + a which conflicts with the readline key mapping for jumping to the beginning of a line. (Side node: C-b also conflicts with an Emacs/readline mapping to move backwards one character. I’ve seen C-q suggested as a safer alternative prefix.)
If you’d rather use something else as a prefix, it’s easy to remap.

Adding the following to your ~/.tmux.conf will remap your prefix to C-a like screen:

unbind C-b
set -g prefix C-a
bind C-a send-prefix

Another default setting you may want to change is the starting window index. Starting with 0 is counterintuitive when it’s at the other end of the keyboard from where the window appears.
To reset the starting window index to 1 use:
set -g base-index 1

Basic Keys

When getting started, the first things you’ll want to know are how to navigate and how get additional help. This should help get you started:

prefix ? — displays the majority of the current key bindings.
prefix c — creates a new window in the current session.
prefix 1 — selects window 1.
prefix 2 — selects window 2, etc.
prefix & — closes the current window after prompting.
prefix " — splits the current pane horizontally in the current window.
prefix % — splits the current pane vertically in the current window.
prefix q — briefly display the ids of the panes in the current window.
prefix o — select next pane (move focus to pane with next id).
prefix x — closes the current pane after prompting.
prefix : — open the tmux command-prompt (tab completion helps for remembering/discovering useful commands)

Customization

One of the best features of tmux is how easy it is to configure. Here are a few things that I’ve used to make my life easier.

Vi-style Key Bindings

# I'm a Vim user, this makes navigation easier
setw -g mode-keys vi # I especially like being able to search with /,? when in copy-mode
unbind-key j
bind-key j select-pane -D # Similar to 'C-w j' to navigate windows in Vim
unbind-key k
bind-key k select-pane -U
unbind-key h
bind-key h select-pane -L
unbind-key l
bind-key l select-pane -R

Mouse Control

If your terminal emulator supports xterm mouse reporting, tmux can also be configured to allow you to select windows, change pane focus, resize panes, and enter copy-mode (scroll-back buffer) using the mouse:

# Just click it
set-option -g mouse-select-pane on
set-option -g mouse-select-window on
set-option -g mouse-resize-pane on

# Scroll your way into copy mode (scrollback buffer)
# and select text for copying with the mouse
setw -g mode-mouse on

Send to all with syncronize-panes

Tmux has a command that will send your input to all panes simultaneously. This is handy when you want to run commands on multiple servers simultaneously.
I toggle this on and off with prefix a:
bind-key a set-window-option synchronize-panes

How to Share Sessions

When confronted with particularly complex and consequential tasks (i.e. refactoring Puppet manifests), Justin and I mitigate risks with ad hoc pairing. Tmux helps us pair on these tough problems by allowing us to easily share our terminal sessions.

Unlike screen which uses setuid and custom ACLs to allow for multiple users¹, tmux utilizes a client/server model with a socket. By specifying a named socket and ensuring the correct permissions, you can allow other users to connect. To make this easier, I have the following aliases defined:

alias tmuxmulti="if tmux -S /tmp/multi has -t AO-PPS; then reattach-to-user-namespace tmux -S /tmp/multi attach; else reattach-to-user-namespace tmux -S /tmp/multi new -s AO-PPS; fi"
alias tmuxer="ssh username@xxx.xxx.xxx.xxx -t '/opt/local/bin/tmux -S /tmp/multi attach'"

Running tmuxmulti will either attach to an existing session or create a new one on the socket /tmp/multi. By ensuring that Justin’s account has read/write access to this socket, I allow him to join the same session. With tmuxer I ssh into Justin’s machine and attach to his tmux session. Generally I have iTerm2 open fullscreen with two vertically split panes. On the left is my workspace and on the right is Justin’s.

Gotchas

When running tmux on OS X, you may encounter a few annoying problems as a result of child processes becoming detached from their parent environments.

Fixing the clipboard on OS X

If you use pbcopy and pbpaste to transfer text from your terminal sessions to the system clipboard, you may find that they do not work from within tmux (or screen or vim). A detailed explanation of why this is can be found here.

TL;DR? Thanks to the work of Paul Hinze you can easily fix it like this:
brew install reattach-to-user-namespace --wrap-pbcopy-and-pbpaste

Losing track of ssh-agent

It’s possible that you may also encounter an issue where new shells lose track of your ssh-agent.
If you encounter this, the annoying result is that you are asked for your password each and every time you need to use your private key (you do have your private key password-protected, right?).

In order to fix this, you need to set an environment variable with the location of the socket that the agent is running on, either by inheriting it from a parent shell’s environment, or rediscovering it.

There are scripts, like this one that will attempt to find ssh-agent’s socket after it has been lost, but I recently discovered that tmux has an option that should save you the trouble. Just add: set -g update-environment -r to your .tmux.conf—it will automatically copy a handful of useful environment variables into new shells from the parent environment.

Color Issues

You may find that things are ugly that weren’t before; for example Vim with a carefully crafted color scheme like Solarized.
Odds are that your terminal type is not being reported correctly. Try adding set -g default-terminal "screen-256color" to your .tmux.conf and see if that helps.

Configuration changes not being loaded

This is a big one that confuses a lot of people. Because tmux uses a client/server model, it only loads .tmux.conf when the server is started, not when a new client attaches. The server will continue running as long as you at least one session running. To manually reload your configuration, you can run :source-file ~/.tmux.conf from the tmux command-line.

Too many sessions

Be careful that you’re not creating a new session each time you launch tmux. It’s easy to pile up old and unused sessions without realizing it. Check what sessions are running on a tmux server by hitting prefix s while in a session or running tmux list-sessions from the command line. If you’re using a named socket you may first specify it like this: tmux -S /tmp/mytmuxsocket list-sessions.

Additional Resources

These blog posts (and countless others) provide some great information on getting started with tmux:

- Practical Tmux

- Vim and tmux on OSX

- Why You Should Try tmux Instead of screen

- Love, hate, & tmux

- A Tmux Crash Course

- Tools I Use – tmux

- TMUX – The Terminal Multiplexer

- TMUX – The Terminal Multiplexer

- What is tmux? – OpenBSD FAQ (recommended by ft on #tmux)

Here’s a link to my current ~/.tmux.conf

¹ I had trouble getting screen’s multi-user mode to work on OS X Lion. There is a bug where screen refuses to allow new clients to attach with a message about having the wrong PID. This is actually what led me to start using tmux.

Back in the day The Well had a text based conference system, you used dial in, then telnet and later ssh to their server and interacted with other members through a text system called PicoSpan. Eventually things moved to the web and it became a lot more forum like. The thing that I really loved was that in the web version of the forums there was a command line. You could type many of the same commands into the web CLI as you would into the Unix one and have the same effects. Posting, searching, jumping through conferences. It was the web with the CLI power for those who wanted it.

The browser is more and more our interface to all things online and frankly it sux a bit, I want the CLI speed for accessing the Web sites that I like. I’ve created a PHP system I called cmd ages ago that simply routed a command like “guk greenwich” to the Google UK search engine with results restricted to those from the UK. There are of course various online tools that does the same but I found that their ‘book’ keyword would search Amazon US while I wanted UK so I just did one that I can tweak to my liking.

Recently thanks to Googles widely hated changes to their Search UI simply redirecting to Google searches with keywords filled in just was not enough anymore. I want web search back the way it was before they made it suck. So I do what hackers do and wrote a Ruby based pluggable search system. You can see a screenshot of it here showing a Google search.

What you’re seeing here is the oldskool-gcse plugin in action. It uses the Google JSON API to query a Google Custom Search Engine and format the results in a way that does not suck. The Custom Search Engines are quite nice as you can customize all sorts of things in them like which sites to exclude, which to favor, limit results to certain countries or languages allowing you to really customize your search experience. The only down side to the GCSE approach is that Google limits API calls to 100 a day, for me that’s enough for searching but ymmv.

Using this method of searching can have some privacy wins, Google recently announced merging all their online accounts into one and will have all your online activity influence your searches. I wasn’t too worried since by then I had already written Oldskool and will simply use a different Google Account to access their search API than the one I use to read my work mail for example. Simple effective win.

My default search in oldskool is a GCSE that resembles a normal Google search but I can also search for “puppet exec” and oldskool will route that request to a specific GCSE that bumps the official Puppet Labs docs to the top, exclude some annoying things etc. So oldskool is a single entry frontend to many different GCSE backends is quite powerful.

As I said it’s plugable and I’ve written one other plugin that uses my Passmakr gem to generate random passwords. I can just search for pass 10 to get a 10 character password:

Writing your own plugins is very easy and I hope to see ones that queries Redmine instances or other internal databases that you might have using the Oldskool framework to display all the data in one handy place.

It retains the most basic feature of simple keyword base redirects, so I can search for book reamde to get Amazon UK book results instantly.

Config is through a simple YAML file:

---
:google_api_key: your.key
:username: http_auth_user
:password: http_auth_pass
:keywords:
- :type: :gcse
  :cx: you_gcse
  :keywords:
  - :default
- :type: :gcse
  :cx: your_gcse
  :keywords:
  - puppet
- :type: :url
  :url: http://amazon.co.uk/exec/obidos/search-handle-url/index=books-uk&field-keywords=%Q%
  :keywords:
  - book
  - books
- :type: :password
  :keywords: pass

This sets up 2 GCSE searches – one marked as my default search – and the mentioned book search and one that uses the password plugin I’ve shown above.

It needs no writable access to the webserver it runs on and it’s all managed by Bundler and Sinatra – perfect for hosting on the free Heroku tier.

As this is effectively my Web CLI I want it integrated in as many places as possible. I use a lot of desktops – 3 regularly – so the browser is my unified UI to all of this. Your instance will publish OpenSearch meta data which will make it seamlessly integrate into Firefox, Chrome, IE, Gnome DO, Gnome Shell and many many other places.

Here’s Firefox search box the first time you browse to a new instance:

And here is Chrome, you do not even have to add it just start typing the URL to your instance and press tab, the URL bar transforms into a Oldskool search box magically. You can add it permanently and make it default by right clicking on the URL bar and choosing Edit Search Engines….

The code is in my GitHub – Oldskool, Oldskool GCSE and Oldskool Password. I will blog again tomorrow or on another day about creating your own plugins etc.

I wrote a post a while back linking to an interesting video about the culture at GitHub, entitled: Optimizing for Happiness – why you want to go work at Github!.

Since then, i’ve watched a few other interesting talks about the culture and how they work at GitHub and two in particular are worth noting here.

Firstly, Zach Holman, one of the early “Githubbers” recently gave a talk about “How GitHub Uses GitHub to Build GitHub“:

Build features fast. Ship them. That’s what we try to do at GitHub. Our process is the anti-process: what’s the minimum overhead we can put up with to keep our code quality high, all while building features as quickly as possible? It’s not just features, either: faster development means happier developers. This talk will dive into how GitHub uses GitHub: we’ll look at some of our actual Pull Requests, the internal apps we build on our own API, how we plan new features, our Git branching strategies, and lots of tricks we use to get everyone – developers, designers, and everyone else involved with new code. We think it’s a great way to work, and we think it’ll work in your company, too.

You can watch the video here and also check out a series of blog posts he wrote on the same subject.

The second talk i’d recommend I had the pleasure of seeing live at a local conference i’ve attend (DIBI Conference). It’s by Corey Donohoe (@atmos):

The talk will cover the metrics driven approach GitHub uses to analyze performance and growth of our product. It will cover deployment strategies for rapid customer feedback as well as configuration management to ensure reproducibility.

You can watch the video here.

Both are great talks and well worth a watch.

In my previous article I mentioned a key strength of Sensu is the ability to re-use existing Nagios plugins. This is a powerful feature of Sensu. Nagios has been around for at least 1000 years according to most recent archaeological discoveries, which means a vast amount of human effort (and capital) has gone into creating Nagios plugins. Being able to leverage this prior effort is a huge win. In this article I’ll demonstrate creating a Sensu check with the check_http Nagios plugin.

Install the check_http nagios plugin

Following on the previous article, we’ll be doing this demo based off of CentOS 5 but it should not be difficult to find a build of check_http for your distribution.

First, Make sure you have the EPEL yum repo installed on your machine. Next, let’s install the nagios-plugin-http package onto our Sensu client node(s):

sudo yum -y install nagios-plugins-http

Let’s see where the check_http binary was installed (note the arch specific path):

$ rpm -ql nagios-plugins-http
/usr/lib64/nagios/plugins/check_http

Write the Sensu check definition

Ok, now we are ready to create a check definition for Sensu. We will create this file on the nodes running sensu-client as well as sensu-server (pro tip: this part is easier if you’re using a CM tool like Chef or Puppet.)

/etc/sensu/conf.d/check_google.json:

{
  "checks": {
    "check_google": {
      "notification": "Google HTTP failed",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins check_http google.com -R 'search'",
      "subscribers": [ "webservers" ],
      "interval": 60,
      "handlers": ["default", "pagerduty"]
    }
  }
}

Let’s look at each part of this check definition:

• notification: This can be thought of as a “friendly message”. It’s most useful with handlers like Twitter or Pagerduty. For example, this will be what you hear when the creepy Pagerduty.com computer voice wakes you up at 3 AM. The full output from the check is also available to handlers in the output attribute.
• command: This is where we specify the check_http command we want to run and relevant options. Notice that we are adding both /usr/lib64/... and /usr/lib/... to the PATH. This is not required, but it is a nice way to make this check work on both i386 and x86_64 platforms for the EPEL-supplied version of check_http.
• subscribers: This is where we specify which sensu-client nodes should run this check. Recall that we defined "subscriptions": "webservers" in our /etc/sensu/conf.d/client.json in the previous article.
• interval: How often this check should be executed.
• handlers: The handlers that should receive the output from this plugin. The handlers will be executed on the sensu-server node.

Turn a profit

After we’ve created the json file on the clients and servers, we need to restart the sensu-client and sensu-server services so they pickup the new .json file.

In a couple minutes, you should see the following in /var/log/sensu/sensu-server.log:

I, [2012-01-23T19:14:22.218301 #7397]  INFO -- : [publisher] -- publishing check request -- check_google -- webservers {"message":"[publisher] -- publishing check request -- check_google -- webservers","level":"info","timestamp":"2012-01-23T19:14:22.   %6N-0700"}

And on the client, in /var/log/sensu/sensu-client.log:

I, [2012-01-23T19:15:22.224437 #18787]  INFO -- : [subscribe] -- received check request -- check_google {"message":"[subscribe] -- received check request -- check_google","level":"info","timestamp":"2012-01-23T19:15:22.   %6N-0700"}
I, [2012-01-23T19:15:22.348884 #18787]  INFO -- : [result] -- publishing check result -- check_google -- 0 -- HTTP OK: HTTP/1.0 200 OK - 11294 bytes in 0.099 second response time |time=0.099463s;;;0.000000 size=11294B;;;0

There we go. We have implemented a Sensu check using a Nagios plugin. There are quite a few Nagios plugins out there and we should be able to use most (all?) of them with Sensu. Go forth and profit.

For questions on using Sensu, stop by the #sensu channel on Freenode.

(5/15/2012) NOTE: This guide has been superseded by the official ‘Install Guide‘ doc on the Sensu wiki. The new process utilizes the simpler Omnibus-style Sensu packages and covers installation on Debian/Ubuntu platforms as well. Please use this guide instead of the instructions below.

I’m excited about Sensu, a new open source monitoring framework, and I’d like to help others get started with it as well. So, after observing the frequent questions from new visitors to #sensu on Freenode I thought perhaps the best way to do that is to write a blog article to help folks get started. If you still have questions after reading this, feel free to come by #sensu on Freenode.

In this article I will provide a brief overview of Sensu with some background, walk through a client and server install, and then I will show you how to add a check and a handler. This should lay the groundwork for future articles with more examples on how to get the most value out of Sensu in your infrastructure.

Before we start, I owe a huge thanks to @jeremy_carroll for the many hours of work he put into building RPM’s for Sensu. His work on packaging will undoubtedly save many folks quite a bit of time.

What is Sensu?

Sensu is the creation of @portertech and his colleagues at sonian.com. They have graciously open-sourced the project and made it available to all of us searching for a modern monitoring platform (or anyone searching for an alternative to Nagios.)

Sensu is often described as the “monitoring router”. Put another way, Sensu connects the output from “check” scripts run across many nodes with “handler” scripts run on Sensu servers. Messages are passed via RabbitMQ. Checks are used, for example, to determine if Apache is up or down. Checks can also be used to collect metrics such as MySQL statistics. The output of checks is routed to one or more handlers. Handlers determine what to do with the results of checks. Handlers currently exist for sending alerts to Pagerduty, IRC, Twitter, etc. Handlers can also feed metrics into Graphite, Librato, etc. Writing checks and handlers is quite simple and can be done in any language.

Key details:

• Ruby 1.8.7+ (EventMachine, Sinatra, AMQP), RabbitMQ, Redis
• Excellent test coverage with continuous integration (travis-ci)
• Messaging oriented architecture. Messages are JSON objects.
• Ability to re-use existing Nagios plugins
• Plugins and handlers (think notifications) can be written in any language
• Supports sending metrics into various backends (Graphite, Librato, etc)
• Designed with modern configuration management systems such as Chef or Puppet in mind
• Designed for cloud environments
• Lightweight, less than 1200 lines of code

Components

Sensu is made up of several small components. I won’t go into too much detail about each of them here, as their purpose will become obvious when we get started, so here a few short descriptions:

sensu-server

The server initiates checks on clients, receives the output of the checks feeds them to handlers. (As of version 0.9.2, clients can also execute checks that the server doesn’t know about and the server will still process their results, more on these ‘standalone checks’ in a future article.)

Sensu-server relies on a Redis instance to keep persistent data. It also relies heavily (as do most sensu components) on access to rabbitmq for passing data between itself and sensu-client nodes.

sensu-client

Run this on all of your systems that you want to monitor. Sensu-client will execute checks scripts (think check_http, check_load, etc) and return the results from these checks to sensu-server via rabbitmq.

sensu-api

A REST API that provides access to various pieces of data maintained on the sensu-server in Redis. You will typically run this on the same server as your sensu-server or Redis instance. It is mostly used by internal sensu components at this time.

sensu-dashboard

Web dashboard providing an overview of the current state of your Sensu infrastructure and the ability to perform actions, such as temporarily silencing alerts.

Installing

As you start to explore Sensu you will find that it was built from the start to be used in conjunction with a CM tool such as Chef or Puppet. However, for the purposes of this article I will walk through a simple manual install.

This article covers installation of Sensu via RPM on CentOS-5 and CentOS-6. Debian/ubuntu and derivatives are not covered in this guide, but many of the same steps will apply. At this time there are no .deb packages for the Sensu components so you will have to install Sensu from gem (ie: gem install sensu sensu-dashboard). Hopefully soon we will have native .deb packages for the Sensu components.

One thing to note when installing from gem: you may get an error when starting sensu-server or sensu-client if you do not have a “client” section defined in your config and if you don’t have at least one “check”.

You will probably want to use Sensu with Chef or Puppet soon after you get bootstrapped (of course you’re already using a modern CM tool in your infrastructure anyway, right?) There are good Chef and Puppet recipes in the github repos that can help you get going fairly quickly. There are also a few community members working on improving these pieces so should get even better over time.

Additionally, the original dev platform for Sensu was Ubuntu but work has been done to help make it a little more CentOS/RHEL-friendly. I’m going to use CentOS 5 and 6 in this article just because I’m more familiar with this platform than the debian/ubuntu family. In any case, it shouldn’t matter too much because the purpose of this article is to show you Sensu.

We will use 2 nodes, one will be our server and the other will be a simple client, with the following bits on each:

Server:

• rabbitmq
• redis
• sensu-server / sensu-client / sensu-api / sensu-dashboard

Client:

• sensu-client

Install a Sensu server node

Install rabbitmq

We will base our approach on the rabbit install guide from here: http://www.rabbitmq.com/install-rpm.html

(CentOS 5 only) We need to install both the EPEL-5 and epel-erlang yum repos. The EPEL-5 yum repo contains the older R12B version of Erlang which would work fine with rabbit except we wouldn’t have access to SSL nor the web management plugins. Thus, we’ll be installing a newer Erlang from the epel-erlang repo which provides R14B for cent5.

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
sudo wget -O /etc/yum.repos.d/epel-erlang.repo http://repos.fedorapeople.org/repos/peter/erlang/epel-erlang.repo

(CentOS 6 only) Install the EPEL-6 yum repo which contains Erlang R14B:

sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Install Erlang:

sudo yum install erlang

Install RabbitMQ:

sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v2.7.1/rabbitmq-server-2.7.1-1.noarch.rpm

We need to make some SSL certs for our rabbitmq server and the sensu clients. I put a simple script up on github to help with this. You’ll want to change a few things in the openssl.cnf to for your organization if you use this in production. The script will generate a few files that we’ll need throughout the guide, so keep them nearby.

git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git
cd joemiller.me-intro-to-sensu/
./ssl_certs.sh clean
./ssl_certs.sh generate

Configure RabbitMQ to use these SSL certs

mkdir /etc/rabbitmq/ssl
cp server_key.pem /etc/rabbitmq/ssl/
cp server_cert.pem /etc/rabbitmq/ssl/
cp testca/cacert.pem /etc/rabbitmq/ssl/

Create /etc/rabbitmq/rabbitmq.config:

[
  {rabbit, [
    {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
                   {verify,verify_peer},
                   {fail_if_no_peer_cert,true}]}
  ]}
].

Install the RabbitMQ webUI management console:

rabbitmq-plugins enable rabbitmq_management

Set RabbitMQ to start on boot and start it up immediately:

sudo /sbin/chkconfig rabbitmq-server on
sudo /etc/init.d/rabbitmq-server start

Verify operation with the RabbitMQ Web UI: Username is “guest”, password is “guest” – http://<SENSU-SERVER>:55672. Protocol amqp should be bound to port 5672 and amqp/ssl on port 5671.

Finally, let’s create a /sensu vhost and a sensu user/password on our rabbit:

rabbitmqctl add_vhost /sensu
rabbitmqctl add_user sensu mypass
rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Install redis

At this point we already have the EPEL repo installed on our server so we will install EPEL’s version of Redis. For Cent5 this will be a fairly old redis v2.0, and for Cent6 it will be v2.2. Both should work fine with Sensu.

sudo yum install redis
sudo /sbin/chkconfig redis on
sudo /etc/init.d/redis start

Install ruby 1.8.7

(CentOS 5 only) Sensu needs ruby 1.8.7+ but CentOS-5 ships with older Ruby 1.8.5. We will use the ruby 1.8.7 rpm’s from Opscode’s Chef. See this page on the Chef wiki for additional details.

sudo wget -O /etc/yum.repos.d/aegisco.repo http://rpm.aegisco.com/aegisco/el5/aegisco.repo

(CentOS 6) CentOS 6 ships with ruby 1.8.7 so we don’t need any external repos.

Install Ruby packages;

sudo yum install ruby ruby-ri ruby-rdoc ruby-shadow rubygems curl openssl-devel

Install Sensu components.

Now we are ready to install Sensu. We will install two rpms: rubygem-sensu and rubygem-sensu-dashboard. This will install four components: sensu-server, sensu-client, sensu-api, sensu-dashboard. Sensu servers use all of these and clients only use sensu-client.

We are going to use @jeremy_carroll‘s recently created yum repo to install Sensu via rpm. This repo contains sensu rpm’s for both CentOS 5 and 6. It’s awesome that Jeremy has taken the time to set this up and I hope we can use this as a basis for automating the building of rpms and debs for all releases of Sensu. Since this repo was setup pretty much as this blog was being written, it’s possible that this repo will move to a different location in the future.

(CentOS 5 only)

sudo rpm -Uvh http://yum.carrollops.com/el/5/sensu-release-1.noarch.rpm

(CentOS 6 only)

sudo rpm -Uvh http://yum.carrollops.com/el/6/sensu-release-1.noarch.rpm

We need to ignore the rubygem rpm’s that come from EPEL because they will conflict with the sensu rpm’s. Edit your /etc/yum.repos.d/epel.repo file and add the following line to the [epel] section.

exclude=rubygem-rack,rubygem-sinatra

Install and enable sensu service components:

sudo yum install rubygem-sensu rubygem-sensu-dashboard
sudo chkconfig sensu-server on
sudo chkconfig sensu-api on
sudo chkconfig sensu-client on
sudo chkconfig sensu-dashboard on

Copy the SSL client key + cert that we created earlier into /etc/sensu/ssl

cp client_key.pem client_cert.pem  /etc/sensu/ssl/

Next we need to configure sensu by editing /etc/sensu/config.json. For now we will create just enough config to start sensu. Later we will add checks and handlers. Note (for later use) that Sensu will also read json config snippets out of the /etc/sensu/conf.d directory so you can piece together a config easily using your CM tool.

{
  "rabbitmq": {
    "ssl": {
      "private_key_file": "/etc/sensu/ssl/client_key.pem",
      "cert_chain_file": "/etc/sensu/ssl/client_cert.pem"
    },
    "port": 5671,
    "host": "localhost",
    "user": "sensu",
    "password": "mypass",
    "vhost": "/sensu"
  },
  "redis": {
    "host": "localhost",
    "port": 6379
  },
  "api": {
    "host": "localhost",
    "port": 4567
  },
  "dashboard": {
    "host": "localhost",
    "port": 8080,
    "user": "admin",
    "password": "secret"
  }
}

Configure /etc/sensu/conf.d/client.json for the current node:

{
  "client": {
    "name": "sensu-server.dom.tld",
    "address": "10.0.0.1",
    "subscriptions": [ "test" ]
  }
}

(CentOS 5 x86_64 only) With the current set of rpm’s we need to add a path to the GEM_PATH in order to find a couple of the rubygems we installed. Run the following:

echo "export GEM_PATH=\$GEM_PATH:/usr/lib64/ruby/gems/1.8" > /etc/profile.d/gem_path.sh
. /etc/profile.d/gem_path.sh

Now let’s try to start the Sensu components:

sudo /etc/init.d/sensu-server start
sudo /etc/init.d/sensu-api start
sudo /etc/init.d/sensu-client start    
sudo /etc/init.d/sensu-dashboard start

If all goes well, the 4 processes mentioned above will be running and the dashboard will be accessible on http://<SENSU SERVER>:8080. Log files are available in /var/log/sensu in case anything is wrong.

sensu    14249  0.0  3.4  92924 17648 ?        S    02:56   0:00 ruby /usr/bin/sensu-server ...
sensu    14404  0.0  4.1 102172 20884 ?        S    03:05   0:00 ruby /usr/bin/sensu-api ...
sensu    14425  0.0  3.7 104860 19292 ?        Sl   03:06   0:00 ruby /usr/bin/sensu-client ...
sensu    14553  0.4  7.0 140544 35932 ?        Sl   03:07   0:00 ruby /usr/bin/sensu-dashboard ...

If you see an openssl error like the one below on CentOS-5, it’s likely because you’re on a x86_64 box but some ruby-libs or ruby-devel 1.8.5 rpm’s from the base repo were accidentally installed, remove them.

Starting sensu-client: /usr/lib/ruby/1.8/openssl/cipher.rb:22: Cipher is not a module (TypeError)
    from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'

Installing a sensu client node

Installing Sensu on a client node is similar to installing the server. We will need to install ruby, the Sensu rpm, and then configure Sensu. There are only a few small differences which are detailed below.

Install ruby 1.8.7

Follow the same steps from the Install ruby 1.8.7 section we used to build our server.

Install sensu-client

Follow the steps from Install Sensu components section we used to build our server, with the following differences:

• Only install rubygem-sensu and skip rubygem-sensu-dashboard
• You will only need the rabbitmq section in /etc/sensu/config.json file. Make sure you point it to your rabbit server.
• Only enable and start the sensu-client service, ie: chkconfig sensu-client on and /etc/init.d/sensu-client start.

The client will log to /var/log/sensu/sensu-client.log.

Add a check

Now that we’ve installed a Sensu server and a client, let’s create a simple check so we can begin to see how the pieces fit together. We’re going to write a check to determine if crond is running. We’ll be using the check-procs.rb script from the sensu-community-plugins repo.

Most of the plugins in the sensu-community-plugins repo rely on the helper classes from the sensu-plugins gem, so let’s install that first. You may need to install gcc in order to build the json gem dependency. Note that we are installing this from a gem because there is not an rpm available yet.

sudo gem install sensu-plugin --no-rdoc --no-ri

Next, we’re going to grab the check-procs.rb script directly from github and install it into /etc/sensu/plugins. You don’t have to install checks into this directory, but it’s convenient.

wget -O /etc/sensu/plugins/check-procs.rb https://raw.github.com/sonian/sensu-community-plugins/master/plugins/processes/check-procs.rb
chmod 755 /etc/sensu/plugins/check-procs.rb

Let’s create a new json file to hold our check definition in /etc/sensu/conf.d/check_cron.json. Put this file on both the Sensu server and client.

(NOTE: as of sensu 0.9.2 ‘standalone’ checks were added which only need to be configured on the client-side. We will cover standalone checks in future articles.)

{
  "checks": {
    "cron_check": {
      "handler": "default",
      "command": "/etc/sensu/plugins/check-procs.rb -p crond -C 1 ",
      "interval": 60,
      "subscribers": [ "webservers" ]
    }
  }
}

Next, we need to tell our client node to listen to subscribe to the webservers queue. The Sensu server will publish a request every 60 seconds on the webservers queue and any client registered to this queue will execute checks that have been registered to this queue. Edit the /etc/sensu/conf.d/client.json file on the client:

{
  "client": {
    "name": "sensu-client.domain.tld",
    "address": "127.0.0.1",
    "subscriptions": [ "test", "webservers" ]
  }
}

Finally, restart sensu on the client and server nodes.

After a few minutes we should see the following in the /var/log/sensu/sensu-client.log on the client:

I, [2012-01-18T21:17:07.561000 #12984]  INFO -- : [subscribe] -- received check request -- cron_check {"message":"[subscribe] -- received check request -- cron_check","level":"info","timestamp":"2012-01-18T21:17:07.   %6N-0700"}

And on the server we should see the following in /var/log/sensu/sensu-server.log:

I, [2012-01-18T21:18:07.559666 #30970]  INFO -- : [publisher] -- publishing check request -- cron_check -- webservers {"message":"[publisher] -- publishing check request -- cron_check -- webservers","level":"info","timestamp":"2012-01-18T21:18:07.   %6N-0700"}
I, [2012-01-18T21:25:07.745012 #30970]  INFO -- : [result] -- received result -- sensu-client.domain.tld -- cron_check -- 0 -- CheckProcs OK: Found 1 matching processes; cmd /crond/

Next, let’s see if we can raise an alert.

/etc/init.d/crond stop

After about a minute we should see an alert on the sensu-dashboard: http://<SERVER IP>:8080

Add a handler

Now that we have created our first check we are ready to hook it up to a handler. Out of the box Sensu ships with a ‘default’ handler which does nothing more than parse the JSON its fed via STDIN and spits back to STDOUT.

There is a growing list of handlers available in the sensu-community-plugins repo, including Pagerduty, IRC, Campfire, etc.

Let’s create a simple handler that simply sends the raw check output to ourselves via email.

The most common handler type is “pipe” which tells Sensu to shell out and run the specified command. We’ll cover more handler types in the future. On the server nodes, we will define our ‘email’ handler in /etc/sensu/conf.d/handler_email.json.

{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "mail -s 'sensu alert' your@address"
    }
  }
}

On the sensu-server and sensu-client nodes we’ll also need to update our check definition and connect it to the new handler, edit the /etc/sensu/conf.d/check_cron.json files and modify the “handlers” attribute:

{
  "checks": {
    "cron_check": {
      "handlers": ["default", "email"],
 ...

Restart sensu-client and sensu-server on the nodes and then stop the crond daemon again. In a few minutes we should get an email from sensu with the subject “sensu alert” and a bag full of JSON data.

This isn’t the most useful handler but it illustrates the concepts of checks and handlers and how they work together. At this point we now have a working sensu-client and sensu-server to start experimenting further. In the future we’ll cover more examples of checks, handlers, metrics, etc.

If you have further questions please visit #sensu on IRC Freenode.

Update (3/2/2012): Olivier Le Cam (@olecam) posted some Debian-specific instructions that accompany this process which also include instructions on building .deb packages for Sensu.

More Sensu articles: sensu