Cuisine : updated & shiny

cuisineIt’s been a while since my last post here, and I have updated my cuisine dashboard yesterday so here is a little follow up on what’s new in it.

First, the asynchronous handler has been updated, pushing backing data a bit differently and fixing a stupid issue on diffs. Check it out, it is needed for this new version of the cuisine dashboard.

I added a consumer that allows to use postgresql as storage backend. This does NOT replace the need for elasticsearch. It’s by making mistakes that you learn, and this one was pretty insightful : it’s better (necessary ?) to have a permanent storage you can rely on. When you come to update mappings, change stuff everywhere you don’t want to loose your data, so reindexing is the solution. There are now 3 scripts to cope with the incoming data :

  • direct indexing to ES, this one does NOT remove data from message queue.
  • storing to PG, data is removed from message queue
  • going to both, data is removed from message queue

These scripts will probably be federated to a single one later, that will be a proper daemon (with logging, pid files and such)

UPDATE : this rewrite is done. A single script allows data consumption, indexing and storage. As a bonus, a quick and dirty packager based on FPM is provided.

In the additions, cuisine now takes enviroments in account, you can filter when looking the last runs and in search you can restrict to a particular one.

To end this post, a few screenshots :

index search

Cuisine : a chef dashboard

cuisine When I wrote the asynchronous chef handler that I presented in the previous post, I had a little idea in mind. being able to track changes made by chef. The idea grew up a little and I now release a little dashboard I wrote. It’s still in a very early stage of development but I’ll try to present the idea behind it.

The changes (including diffs) are pushed in a queue. This queue is consumed by a script and datas are indexed in elasticsearch, an open source search engine. On the top of this I wrote a web interface, based on sinatra and twitter’s bootstrap) that allow you to see the latest runs, filter out runs with no changes and search on criterias (hostname, updated resources and inside the diffs)

To use this you will need a couple of things :

  • a STOMP broker (I use rabbitmq, but activemq or stompserver will fit too)
  • an elasticsearch instance (or cluster)
  • sinatra and its dependencies + the stomp ruby gem

The code is available on github, feel free to get in touch on freenode IRC, you can find me on the #chef-hacking channel (nickname : nico)

Asynchronous reporting with chef

gousset Configuration management tools are awesome. Using them, you are now managing loads of servers, reaching the pub on time and you can focus on really fun stuff. A counterpart is that they almost work in your back : changes are propagated quickly, and even if you store your cookbooks/modules in a VCS, even if you review them, you still want to know what really happens on your servers. Puppet has a really nice automatic summary part to do that job, and some people are doing cool things with it. Chef also has summary of updated resources but they don’t get back to a master like puppet do. So I wrote a little report handler to push back data to wherever you want to. It is based on the stomp protocol, to be non blocking and easily scale if you have a large number of machine. With this you just need to have a consumer that will process data to your favorite datastore.

I’ve put that on my github chef repo, check the readme for the extensive file list & function.