Baptiste Daroussin did an incredible job on FreeBSD with the new packages system, named PkgNG. It brings modern workflow, options and shiny features that were needed for a long time. Say goodbye to painfully long upgrades.
However, Chef is not yet able to use this packaging system as it does not have a PkgNG provider, or not had. This is a hacky way to do so but here is a way to use PkgNG with chef, making it the default provider for your packages.
It’s been a while since my last post here, and I have updated my cuisine dashboard yesterday so here is a little follow up on what’s new in it.
First, the asynchronous handler has been updated, pushing backing data a bit differently and fixing a stupid issue on diffs. Check it out, it is needed for this new version of the cuisine dashboard.
I added a consumer that allows to use postgresql as storage backend. This does NOT replace the need for elasticsearch. It’s by making mistakes that you learn, and this one was pretty insightful : it’s better (necessary ?) to have a permanent storage you can rely on. When you come to update mappings, change stuff everywhere you don’t want to loose your data, so reindexing is the solution. There are now 3 scripts to cope with the incoming data :
direct indexing to ES, this one does NOT remove data from message queue.
storing to PG, data is removed from message queue
going to both, data is removed from message queue
These scripts will probably be federated to a single one later, that will be a proper daemon (with logging, pid files and such)
UPDATE : this rewrite is done. A single script allows data consumption, indexing and storage. As a bonus, a quick and dirty packager based on FPM is provided.
In the additions, cuisine now takes enviroments in account, you can filter when looking the last runs and in search you can restrict to a particular one.
When I wrote the asynchronous chef handler that I presented in the previous post, I had a little idea in mind. being able to track changes made by chef. The idea grew up a little and I now release a little dashboard I wrote. It’s still in a very early stage of development but I’ll try to present the idea behind it.
The changes (including diffs) are pushed in a queue. This queue is consumed by a script and datas are indexed in elasticsearch, an open source search engine. On the top of this I wrote a web interface, based on sinatra and twitter’s bootstrap) that allow you to see the latest runs, filter out runs with no changes and search on criterias (hostname, updated resources and inside the diffs)
To use this you will need a couple of things :
a STOMP broker (I use rabbitmq, but activemq or stompserver will fit too)
an elasticsearch instance (or cluster)
sinatra and its dependencies + the stomp ruby gem
The code is available on github, feel free to get in touch on freenode IRC, you can find me on the #chef-hacking channel (nickname : nico)
Configuration management tools are awesome. Using them, you are now managing loads of servers, reaching the pub on time and you can focus on really fun stuff. A counterpart is that they almost work in your back : changes are propagated quickly, and even if you store your cookbooks/modules in a VCS, even if you review them, you still want to know what really happens on your servers. Puppet has a really nice automatic summary part to do that job, and some people are doing cool things with it. Chef also has summary of updated resources but they don’t get back to a master like puppet do. So I wrote a little report handler to push back data to wherever you want to. It is based on the stomp protocol, to be non blocking and easily scale if you have a large number of machine. With this you just need to have a consumer that will process data to your favorite datastore.
I’ve put that on my github chef repo, check the readme for the extensive file list & function.
Mcollective has been able to fire up puppetd runs for a while now, via a standalone RPC call or through the puppet commander binary (check it out, spread your load). I wanted to be able to fire up my chef clients with mcollective, to use metadata to filter what should be impacted. So I wrote a little piece of ruby, mostly based on the puppet one, to achieve this. You can now do the following :
It started like a toy, to learn a little more about mcollective agents but I finally turned into something useful (at least for me). I pushed my agent “smith” on my github account. It allows you to install or remove agents within mcollective. I usually use my configuration management tool to deploy such pieces of software but it can be useful in some case to go without it.
The mandatory internet meme reference : yes xzibit approves.
This post follows my previous one, dealing with the reuse of chef providers of chef in mcollective. In the comments Adam Jacob had an interesting word and when I wrote my second agent, to manage package I saw it would be a piece of cake to write a really generic agent, due to the nature of chef resource (and the way to invoke them)
So, this is a generic chef resource mcollective agent, with the associated example client code. It anyway deserves an little explanation; it is not mean to work with a command line invocation. Why ? Because I push quite “complex” data as the resourceactions parameter. The only way I found to make this work from command line is to use eval on the argument, which is no way acceptable. Anyway I hope some people will find this useful.
It has been quite calm for a couple of months here. I have switched job, it explains why I had less time to post some things.I now work at fotolia, and I switched from puppet to chef (no troll intended, I still think puppet is a great tool, please read this).
However, a tool I still have is the awesome mcollective. Unfortunately, the most used agents (package, service) relay on puppet providers to do their actions. Fortunately, open source is here, so I wrote a (basic) service agent that uses chef providers to start/stop or restart an agent. It still needs some polish for the status part (ho the ugly hardcoded path) but I was quite excited to share this. Freshly pushed on github !
Thanks to Jordan Sissel for minstrel, an awesome debug tool, the opscode team for the help on the provider and R.I. Pienaar for mcollective (and the support).
Basing more and more stuff on mcollective means relying more and more on one of its underlying components : the activeMQ middleware, and more precisely the stomp connector. I hit a weird bug a few days ago and realized that I was not functionnaly monitoring this part of the system. The port was bound and responded to connections, subscriptions were possible but messages didn’t pass through.So I wrote this little plugin that makes this possible : it creates a random string, sends it to a queue and then reads the queue to check if the result is the same.
This has been possible with the help of @ripienaar. Thanks for the explanation for the difference between topics & queues !