Lies, Damn Lies and Statistics, 2016 Edition

When people sign up for Configuration Management Camp, we ask them what community room they are mostly interested in.
We ask this question because we have rooms in different sizes and we don't want to put communities with 20 people showing interest in a 120 seat room and we don't want to put a community with 200 people in a 60 seat room.

But it also gives us to opportunity to build some very interesting graph over the potential evolution of the communities.

So looking at the figures ... the overall community is obviously growing,From 350 to 420, to just short of 600 people registered now.

The Puppet Community is not the biggest anymore, that spot went to the Ansible Community room. And all but the CFengine communities are growing.

One more thing , The organisation team discussed several times if we should rebrand the event. We opted not to .. Infracoders.eu could have been an alternative name .. but we decided to stick with the name that already is known,
the content will evolve.. but Config Management Camp will stay the place where people that care about Infrastructure as Code and Infrastructure automation meet.

Lies, Damn Lies and Statistics, 2016 Edition

When people sign up for Configuration Management Camp, we ask them what community room they are mostly interested in.
We ask this question because we have rooms in different sizes and we don't want to put communities with 20 people showing interest in a 120 seat room and we don't want to put a community with 200 people in a 60 seat room.

But it also gives us to opportunity to build some very interesting graph over the potential evolution of the communities.

So looking at the figures ... the overall community is obviously growing,From 350 to 420, to just short of 600 people registered now.

The Puppet Community is not the biggest anymore, that spot went to the Ansible Community room. And all but the CFengine communities are growing.

One more thing , The organisation team discussed several times if we should rebrand the event. We opted not to .. Infracoders.eu could have been an alternative name .. but we decided to stick with the name that already is known,
the content will evolve.. but Config Management Camp will stay the place where people that care about Infrastructure as Code and Infrastructure automation meet.

Lies, Damn Lies and Statistics, 2016 Edition

When people sign up for Configuration Management Camp, we ask them what community room they are mostly interested in.
We ask this question because we have rooms in different sizes and we don't want to put communities with 20 people showing interest in a 120 seat room and we don't want to put a community with 200 people in a 60 seat room.

But it also gives us to opportunity to build some very interesting graph over the potential evolution of the communities.

So looking at the figures ... the overall community is obviously growing,From 350 to 420, to just short of 600 people registered now.

The Puppet Community is not the biggest anymore, that spot went to the Ansible Community room. And all but the CFengine communities are growing.

One more thing , The organisation team discussed several times if we should rebrand the event. We opted not to .. Infracoders.eu could have been an alternative name .. but we decided to stick with the name that already is known,
the content will evolve.. but Config Management Camp will stay the place where people that care about Infrastructure as Code and Infrastructure automation meet.

2014 vs 2015 interest in Open Source Configuration Management

A couple of people asked me to results of the survey of the 2015 vs 2014 Configuration Management Camp room interrests.

This is a bunch of 350 last year and 420 people telling us what tools they are interested in so we can map the right roomsizes to the communities.

2014 :

2015:

Enjoy.. but remember there's Lies, Damn Lies and Statistics ..
PS. this is a mostly European Audience .

Command-line cookbook dependency solving with knife exec

Note: This article was originally published in 2011. In response to demand, I've updated it for 2014! Enjoy! SNS

Imagine you have a fairly complicated infrastructure with a large number of nodes and roles. Suppose you have a requirement to take one of the nodes and rebuild it in an entirely new network, perhaps even for a completely different organization. This should be easy, right? We have our infrastructure in the form of code. However, our current infrastructure has hundreds of uploaded cookbooks - how do we know the minimum ones to download and move over? We need to find out from a node exactly what cookbooks are needed for that node to be built.

The obvious place to start is with the node itself:

$ knife node show controller
Node Name:   controller
Environment: _default
FQDN:        controller
IP:          182.13.194.41
Run List:    role[base], recipe[apt::cacher], role[pxe_server]
Roles:       pxe_server, base
Recipes      apt::cacher, pxe_dust::server, dhcp, dhcp::config
Platform:    ubuntu 10.04

OK, this tells us we need the apt, pxe_dust and dhcp cookbooks. But what about them - do they have any dependencies? How could we find out? Well, dependencies are specified in two places - in the cookbook metadata, and in the individual recipes. Here's a primitive way to illustrate this:

bash-3.2$ for c in apt pxe_dust dhcp
> do
> grep -iER 'include_recipe|^depends' $c/* | cut -d '"' -f 2 | sort | uniq
> done
apt::cacher-client
apache2
pxe_dust::server
tftp
tftp::server
utils

As I said - primitive. However the problem doesn't end here. In order to be sure, we now need to repeat this for each dependency, recursively. And of course it would be nice to present them more attractively. Thinking about it, it would be rather useful to know what cookbook versions are in use too. This is definitely not a job for a shell one liner - is there a better way?

As it happens, there is. Think about it - the Chef server already needs to solve these dependencies to know what cookbooks to push to API clients. Can we access this logic? Of course we can - clients carry out all their interactions with the Chef server via the API. This means we can let the server solve the dependencies and query it via the API ourselves.

Chef provides two powerful ways to access the API without having to write a RESTful client. The first, Shef, is an interactive REPL based on IRB, which when launched gives access to the Chef server. This isn't trivial to use. The second, much simpler way is the knife exec subcommand. This allows you to write Ruby scripts or simple one-liners that are executed in the context of a fully configured Chef API Client using the knife configuration file.

Now, since I wrote this article, back in summer 2011, the API has changed, which means that my original method no longer works. Additionally, we are now served by at least two local dependency solvers, in the form of Berkshelf (whose dependency solver, 'solve' is now available as an individual Gem), and Librarian-chef. In this updated version, I'll show how to use the new Chef server API to perform the same function. Berkshelf and Librarian solve a slightly different problem, in that in this instance we're trying to solve dependencies for a node, so for the purposes of this article I'll consider them out of scope.

For historical purposes, here's the original solution:

knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'

The /nodes/NODE_NAME/cookbooks endpoint returns the cookbook attributes, definitions, libraries and recipes that are required for this node. The response is a hash of cookbook name and Chef::CookbookVersion object. We simply iterate over each one, and pretty print the cookbook name and the version.

Let's give it a try:

$ knife exec -E '(api.get "nodes/controller/cookbooks").each { |cb| pp cb[0] => cb[1].version }'
{"apt"=>"1.1.1"}
{"tftp"=>"0.1.0"}
{"apache2"=>"0.99.3"}
{"dhcp"=>"0.1.0"}
{"utils"=>"0.9.5"}
{"pxe_dust"=>"1.1.0"}

The current way to solve dependencies using the Chef server API resides under the environments end point. This makes sense, if you think of environments as a way to define and constrain version numbers for a given set of nodes. This means that constructing the API call, and handling the results is slightly more than can easily be comprehended in a one-liner, which gives us the opportunity to demonstrate the use of knife exec with a script on the filesystem.

First let's create the script:

USAGE = "knife exec script.rb NODE_NAME"

def usage_and_exit
  STDERR.puts USAGE
  exit 1
end

node_name = ARGV[2]

usage_and_exit unless node_name

node = api.get("nodes/#{node_name}")
run_list_expansion = node.expand!("server")

cookbook_solution = api.post("environments/#{node.chef_environment}/cookbook_versions",
                            :run_list => run_list_expansion.recipes)

cookbook_solution.each do |name, cb|
  puts name + " => " + cb.version
end

exit

The way knife exec scripts work is to pass the arguments following knife to Ruby as the ARGV special variable, which is an array of each space-separated argument. This allows us to produce a slightly more general solution, to which we can pass the name of the node for which we want to solve. The usage handling is obvious - we print the usage to stderr if the command is called without a node name. The meat of the script is the API call. First we get the node object (from ARGV[2], i.e. the node we passed to the script) from the Chef server. Next we expand the run list - this means check for and expand any run lists in roles. Finally we call the API to provide us with cookbook versions for the specified node in the environment in which the node currently resides, passing in the recipes from the expanded run list. Finally we iterate over the cookbooks we get back, and print the name and version. Note that this script could easily be modified to solve for a different environment, which would be handy if we wanted to confirm what versions we'd get were we to move the node to a different environment. Let's give in a whirl:

$ knife exec src/knife-cookbook-solve/solve.rb asl-dev-1
chef_handler => 1.1.4
minitest-handler => 0.1.3
base => 0.0.2
hosts => 0.0.1
yum => 2.3.0
tmux => 1.1.1
ssh => 0.0.6
fail2ban => 1.2.2
users => 2.0.6
security => 0.1.0
sudo => 2.0.4
atalanta-users => 0.0.2
community_users => 1.5.1
sudoersd => 0.0.2
build-essential => 1.4.2

To conclude as did the original article....Nifty! :)

Using Test Doubles in ChefSpec

One of the improvements to ChefSpec in the 3.0 release is the ability to extend test coverage to execute blocks. Doing so requires the infrastructure developer to stub out shell commands run as part of the idempotence check. This is pretty simple as ChefSpec provides a macro to stub shell commands. However doing so where the idempotence check uses a Ruby block is slightly more involved. In this article I explain how to do both.

Quick overview of ChefSpec

ChefSpec is a powerful and flexible unit testing utility for Chef recipes. Extending the popular Ruby testing tool, Rspec, it allows the developer to make assertions about the Chef resources declared and used in recipe code. The key concept here is that of Chef resources. When we write Chef code to build infrastructure, we're using Chef's domain-specific language to declare resources - abstractions representing the components we need to build and configure. I'm not going to provide a from-the-basics tutorial here, but dive in with a simple example. Here's a test that asserts that our default recipe will use the package resource to install OpenJDK.

require 'chefspec'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

If we run this first, we'll see something like this:

$ rspec -fd spec/default_spec.rb 

stubs-and-doubles

================================================================================
Recipe Compile Error
================================================================================

Chef::Exceptions::RecipeNotFound
--------------------------------
could not find recipe default for cookbook stubs-and-doubles

  installs OpenJDK (FAILED - 1)

Failures:

  1) stubs-and-doubles installs OpenJDK
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Chef::Exceptions::RecipeNotFound:
       could not find recipe default for cookbook stubs-and-doubles
     # ./spec/default_spec.rb:10:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:13:in `block (3 levels) in <top (required)>'

Finished in 0.01163 seconds
1 example, 1 failure

Failed examples:

rspec ./spec/default_spec.rb:12 # stubs-and-doubles installs OpenJDK

This is reasonable - I've not even written the recipe yet. Once I add the default recipe, such as:

package 'java-1.7.0-openjdk'

Now the test passes:

$ rspec -fd spec/default_spec.rb 


stubs-and-doubles
  installs OpenJDK

Finished in 0.01215 seconds
1 example, 0 failures

Test Doubles

ChefSpec works by running a fake Chef run, and checking that the resources were called, with the correct parameters. Behind the scenes, your cookbooks are loaded, but instead of performing real actions on the system, the Chef Resource class is modified such that messages are sent to ChefSpec instead. One of the key principles of Chef is that resources should be idempotent - the action should only be taken if required, and it's safe to rerun the resource. In most cases, the Chef provider knows how to guarantee this - it knows how to check that a package was installed, that a directory was created. However, if we use an execute resource - a resource where we're calling directly to the underlying operating system - Chef has no way to tell if the command we called did the right thing. Unless we explicitly tell Chef how to check, it will just run the command again and again. This causes a headache for ChefSpec, because it doesn't have a built-in mechanism for faking operating system calls - so when it comes across a guard, it requires us to help it out, by stubbing the command.

This introduces some testing vocabulary - vocabulary that is worth stating explicitly, for the avoidance of confusion. I'm a fan of the approach described in Gerard Meszaros' 2007 book xUnit Test Patterns: Refactoring Test Code, and this is the terminology used by Rspec. Let's itemise a quick glossary:

  • System Under Test (SUT) - this is the thing we're actually testing. In our case, we're testing the resources in a Chef recipe. Note we're explictly not testing the operating system.
  • Depended-on Component (DOC) - usually our SUT has some external dependency - a database, a third-party API, or on our case, the operating system. This is an example of a DOC
  • Test Double - when unit testing, we don't want to make real calls to the DOC. It's slow, can introduce unwanted variables into our systems, and if it becomes unavailable our tests won't run, or will fail. Instead we want to be able to interact with something that represents the DOC. The family of approaches to implement this abstraction is commonly referred to as Test Doubles.
  • Stubbing - when our SUT depends on some input from the DOC, we need to be able to control those. A typical approach is to stub the method that makes a call to the DOC, typically returning some canned data.

Let's look at a real example. The community runit cookbook, when run on a RHEL derivative, will, by default, build an RPM and install it. The code to accomplish this looks like this:

bash 'rhel_build_install' do
      user 'root'
      cwd Chef::Config[:file_cache_path]
      code <<-EOH
tar xzf runit-2.1.1.tar.gz
cd runit-2.1.1
./build.sh
rpm_root_dir=`rpm --eval '%{_rpmdir}'`
rpm -ivh '/root/rpmbuild/RPMS/runit-2.1.1.rpm'
EOH
      action :run
      not_if rpm_installed
end

Observe the guard - not_if rpm_installed. Earlier in the recipe, that method is defined as:

rpm_installed = "rpm -qa | grep -q '^runit'"

ChefSpec can't handle direct OS calls, and so if we include the runit cookbook in our recipe, we'll get an error. Let's start by writing a simple test that asserts that we include the runit recipe. I'm going to use Berkshelf as my dependency solver, which means I need to add a dependency in my cookbook metadata, and supply a Berksfile that tells Berkshelf to check the metadata for dependencies. I also need to add Berkshelf support to my test. My test now looks like this:

require 'chefspec'
require 'chefspec/berkshelf'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'includes the runit recipe' do
      expect(chef_run).to include_recipe 'runit'
    end

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

And my recipe like this:

include_recipe 'runit'
package 'java-1.7.0-openjdk'

Now, when I run the test, ChefSpec complains:

1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     ChefSpec::Error::CommandNotStubbed:
       Executing a real command is disabled. Unregistered command: `command("rpm -qa | grep -q '^runit'")`"

       You can stub this command with:

         stub_command("rpm -qa | grep -q '^runit'").and_return(...)
     # ./spec/default_spec.rb:11:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:14:in `block (3 levels) in <top (required)>'

ChefSpec tells us exactly what we need to do, but let's unpack it a little, using the vocabulary from above. The SUT, our stunts-and-doubles cookbook, has a dependency on the operating system - the DOC. This means we need to be able to insert a test double of the operating system, specifically a test stub, which will provide a canned answer to our rpm command. ChefSpec makes it very easy for us by providing a macro that does exactly this. We need to run this before every example, so we can put it in a before block. The new test now looks like this:

require 'chefspec'
require 'chefspec/berkshelf'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    before(:each) do
      stub_command("rpm -qa | grep -q '^runit'").and_return(true)
    end

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'includes the runit recipe' do
      expect(chef_run).to include_recipe 'runit'
    end

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

Now when we run the test, it passes:

$ rspec -fd spec/default_spec.rb 

stubs-and-doubles
  includes the runit recipe
  installs OpenJDK

Finished in 0.57793 seconds
2 examples, 0 failures

That's all fine and dandy, but suppose we execute some Ruby for our guard instead of a shell command. Here's an example from one of my cookbooks, in which I set the correct Selinux policy to allow apache to proxy to a locally running Netty server:

unless (node['platform'] == 'Amazon' or node['web_proxy']['selinux'] == 'Disabled')
  execute 'Allow Apache Network Connection in SELinux' do
    command '/usr/sbin/setsebool -P httpd_can_network_connect 1'
    not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }
    notifies :restart, 'service[httpd]'
  end
end

Now, OK, I could have used grep, but I prefer this approach, and it's a good enough example to illustrate how we handle this kind of case in ChefSpec. First, let's write a test:

it 'sets the Selinux policy to allow proxying to localhost' do
  expect(chef_run).to run_execute('Allow Apache Network Connection in SELinux')
  resource = chef_run.execute('Allow Apache Network Connection in SELinux')
  expect(resource).to notify('service[httpd]').to(:restart)
end

If we were to run this, ChefSpec would complain that we didn't have an execute resource with a run action on our run list. So we then add the execute block from above to the default recipe. I'm going to omit the platform check for simplicity, and just include the execute resource. We're also going to need to define an httpd service. Of course we're never going to actually run this code, so I'm not fussed that the service exists despite us never installing Apache. My concern in this article is to teach you about the testing, not write a trivial and pointless cookbook.

Now our recipe looks like this:

include_recipe 'runit'
package 'java-1.7.0-openjdk'

service 'httpd'

execute 'Allow Apache Network Connection in SELinux' do
  command '/usr/sbin/setsebool -P httpd_can_network_connect 1'
  not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }
  notifies :restart, 'service[httpd]'
end

When we run the test, we'd expect all to be fine. We're asserting that there's an execute resource, that runs, and that it notifies the httpd service to restart. However, this is what we see:

Failures:

  1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:23:in `block (3 levels) in <top (required)>'

  2) stubs-and-doubles installs OpenJDK
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:27:in `block (3 levels) in <top (required)>'

  3) stubs-and-doubles sets the Selinux policy to allow proxying to localhost
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:31:in `block (3 levels) in <top (required)>'

Finished in 1.11 seconds
3 examples, 3 failures

Boom! What's wrong? Well, ChefSpec isn't smart enough to warn us about the guard we tried to run, and actually tries to run the Ruby block. I'm (deliberately) running this on a machine without the ability to run the getsebool command to trigger this response, but on my usual workstation running Fedora, this will silently pass. This is what prompted me to write this article, because my colleague who runs these tests on his mac kept getting this No such file or directory - getsebool httpd_can_network_connect error, despite the Jenkins box (running CentOS) and my workstation working just fine. So - what's the solution? Well, we need to do something similar to that which ChefSpec did for us earlier. We need to create a test double, only this time it's Mixlib::ShellOut that we need to stub. There are three steps we need to follow. We need to capture the :new method that is called on Mixlib::ShellOut, and instead of returning canned data, like we did when we called stub_command, we want to return the test double, standing in for the real instance of Mixlib::Shellout, and finally we want to control the behaviour of the test double, making it return the output we want for out test. So, first we need to create the test double. We do that with the double method in Rspec:

shellout = double

This just gives us a blank test double - we can do anything we like with it. Now we need to stub the constructor, and return the double:

Mixlib::ShellOut.stub(:new).and_return(shellout)

Finally, we specify how the shellout double should respond when it receives the :run_command method.

  allow(shellout).to receive(:run_command).and_return('--> off')

We want the double to return a string that won't cause the guard to be triggered, because we want to assert that the execute method is called. We can add these three lines to the before block:

before(:each) do
  stub_command("rpm -qa | grep -q '^runit'").and_return(true)
  shellout = double
  Mixlib::ShellOut.stub(:new).and_return(shellout)
  allow(shellout).to receive(:run_command).and_return('--> off')
end

Now when we run the test, we'd expect the Mixlib guard to be stubbed, the test double returned, and the test double to respond to having the :run_command method called be that it returns a string which doesn't match the guard, and thus the execute should run! Let's give it a try:

Failures:

  1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     NoMethodError:
       undefined method `stdout' for "--> off":String
     # /tmp/d20140208-30741-eynz5u/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:23:in `block (3 levels) in <top (required)>'

Alas! What have we done wrong? Look closely at the error. Ruby tried to call :stdout on a String. Why did it do that? Look at the guard again:

not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }

Aha... we need another double. When the first double is called, we need to return something that can accept a stdout call, which in turn will return the string. Let's add that in:

before(:each) do
  stub_command("rpm -qa | grep -q '^runit'").and_return(true)
  shellout = double
  getsebool = double
  Mixlib::ShellOut.stub(:new).and_return(shellout)
  allow(shellout).to receive(:run_command).and_return(getsebool)
  allow(getsebool).to receive(:stdout).and_return('--> off')
end

Once more with feeling:

$ bundle exec rspec -fd spec/default_spec.rb 

stubs-and-doubles
  includes the runit recipe
  installs OpenJDK
  sets the Selinux policy to allow proxying to localhost

Finished in 0.7313 seconds
3 examples, 0 failures

Just to illustrate how the double interacts with the test, let's quickly change what getsebool returns:

allow(getsebool).to receive(:stdout).and_return('--> on')

Now when we rerun the test, it fails:

Failures:

  1) stubs-and-doubles sets the Selinux policy to allow proxying to localhost
     Failure/Error: expect(chef_run).to run_execute('Allow Apache Network Connection in SELinux')
       expected "execute[Allow Apache Network Connection in SELinux]" actions [] to include :run
     # ./spec/default_spec.rb:31:in `block (3 levels) in <top (required)>'

This time the guard prevented the execute from running, and as such the resource collection didn't contain this resource, and so the test failed.

Conclusion

One of the great beauties of ChefSpec (and of course Chef) is that at its heart it's just Ruby. This means that at almost any point you can reach into the standard Ruby development toolkit for your testing or infrastructure development needs. Hopefully this little example will be helpful to you. If it inspires you to read more about mocking, I can recommend the following resources:

FOSDEM 2014 is coming

and with that almost a full week of side events.
For those who don't know FOSDEM, (where have you been hiding for the past 13 years ? ) Fosdem is the annual Free and Open Source Developers European meeting. If you are into open source , you just can't mis this event where thousands of likeminded people will meet.

And if 2 days of FOSDEM madness isn't enough people organise events around it.

Last year I organised PuppetCamp in Gent, the days before Fosdem and a MonitoringLove Hackfest in our office the 2 days after FOSDEM This year another marathon is planned.

On Friday (31/1/2014) the CentOs community is hosting a Dojo in Brussels at the IBM Forum. (Free, but registration required by the venue)

After the success of PuppetCamp in Gent last year we decided to open up the discussion and get more Infrastructure as Code people involved in a CfgMgmtCamp.eu

The keynotes for CfgMgmtCamp will include the leaders of the 3 most popular tools around , both Mark Burgess, Luke Kanies and Adam Jacob will present at the event which will take place in Gent right after Fosdem. We expect people from all the major communities including, but not limited to , Ansible, Salt, Chef, Puppet, CFengine, Rudder, Foreman and Juju (Free but registration required for catering)

And because 3 events in one week isn't enough the RedHat Community is hosting their Infrastructure.next conference after CfgMgmtCamp at the same venue. (Free but registration required for catering)

cya in Belgium next year..

Chef: The Definitive Guide

Writing books is hard. I used to think it was a lot like writing a blog, only scaled up a bit. But it's much more involved than that. Once you're a published author, suddenly you have obligations and responsibilities to your publisher, and to the paying, reading public. The ante has been well and truly upped.

A few years ago I wrote a slim volume - Test-driven infrastructure with Chef. At the time I wrote it, I was pretty much the only person trying to do infrastructure code in a fashion inspired by the TDD and BDD schools of thought which I practiced as a software developer. When it was published, it was remarkably popular, and despite really being a book about test-driven development, and infrastructure code, it was widely read as a book 'about' Chef.

The problem was, this was the only published book on Chef. Chef as a tool was growing in popularity, and regular complaints were heard about the quality of the public documentation on the Opscode wiki, and about the failure of my volume to be a comprehensive introduction to Chef. Notwithstanding the observation that my first book was never intended to be a comprehensive introduction to Chef, both O'Reilly and I took this on board, and began work on a "Definitive Guide" - a full-length, comprehensive book on Chef. And that's where the problems started.

Chef is a large project, with a very active community. It's also a young project, moving very quickly indeed. Any attempt to capture the 'recommended approach' at a point in time was quickly rendered obsolete. New features were being added at breakneck speed. New community tools were being developed, and 'best practice' was in a constant state of flux. It was clear that writing a 'definitive' guide was going to be a significant undertaking, and quite possibly a flawed one.

At the same time, interest in test-driven infrastructure was exploding. New testing approaches and tools were blossoming, dozens of talks and discussions were being had, and my little introduction to the idea was getting slammed by its readership for covering only one approach, for being too short, and for not being a definitive guide to Chef. Did I mention that writing books is hard?

I had to make a decision. With limited time available, with a busy speaking, training and consulting schedule, and a large family, where should I focus my attention? After discussions with O'Reilly, and friends and colleagues, I decided that I should work on updating the initial Test-driven Infrastructure book. Now, in microcosm, we had the very same problem of a rapidly growing toolset and community to deal with. Tools came and went, underwent complete rewrites, and best practices continued to evolve. I also felt it was necessary to try to give a more thorough Chef overview, in response to feedback on the first volume. I worked incredibly hard on the 2nd edition. Frankly, too hard. I made myself ill, got entirely burned out, and let down family, friends, colleagues and readers. But, by the time of Velocity, this summer, we were pretty much finished. I met with my editor at the conference, and we surveyed the landscape.

The much-hated wiki had been replaced by a greatly improved documentation website. Ospcode had hired a technical writer to work on documentation, and community engagement in improving and maintaining that resource was growing. We remained convinced that trying to write a "definitive guide" at this stage was not a wise choice.

Additionally, Seth Vargo had begun an excellent project: learnchef. Aimed at being the ultimate hands-on quickstart guide, this was immediately well-received, and together with my second edition, filled the requirement for an introduction to Chef. The intermediate, reference-level specification of the core Chef functionality was adequately covered by docs.opscode.com. What we felt was missing was deep-dive, subject-specific discussions. How do I build infrastructure on Windows? How can I build a continuous delivery pipeline using Chef? Advanced programming in the Chef environment. That sort of thing.

We agreed to cancel the definitive guide project, with a view to working on these subject-specific guides. I tweeted about this, more than once, and shared our intention with friends in the community. What we didn't do, either O'Reilly, or me, was make a formal, public announcement. That was a mistake. I can't apologise on behalf of O'Reilly, but I can apologise personally. That was unprofessional of me: I'm sorry.

So, it's now summer, and I'm utterly exhausted. But in the spirit of the invincible superhero, I continued to take on work, travel, speak, and generally over-commit. My physical and mental health deteriorated further, until late August when I had to accept I was at the point of complete breakdown. I took about 6 weeks off, recovered my perspective, got some good rest, and left the editorial process in the safe hands of O'Reilly, and Helena.

Fast-forward to now. The 2nd edition of Test-Driven Infrastructure is out, and early reviews are positive. I'm rested, healthy, and hopefully wiser. I've learned a lot about Chef, and about writing, and am ready to start on my next project... this time with co-authors from day one. I have ideas on what we should cover first, but I'm open to suggestions and requests.

In the meantime, we have good resources available. Use learnchef, join the IRC channels, participate in the mailing list, read my book. Matthias Marschall has just finished his book on Chef, which is also excellent. People who lament the quality of the official documentation - please: give specific examples of where you feel information is missing, the writing is poor, the material is misleading. Remember: this is a community project - if you think you can improve the documentation, submit a pull request, and make it better for everyone. Opscode is committed to great documentation, and the decision not to try to write a definitive guide forces us as a community to build this reference ourselves, openly.

To conclude - I acknowledge that I've let down the people who were so eagerly anticipating "The Definitive Guide". I also accept that we handled the communication of our decision badly. But I think it's the right decision. And I think we're in a strong position to move forward and build on the resources we already have as a community. Will you help?

Tooling for an Iterative Infrastructure

Iteration is valuable to more than just application development; design also greatly benefits. But the benefits don’t stop there — infrastructure can gain as well.

A big portion of the value in an iterative approach comes from the empirical evidence gained through the experience of using the software. By exposing your product to the light of reality, all of your theories are put to the test. This feedback loop ensures that each cycle of work stays tightly focused on the things that really matter — even when they change and especially when they turn out to be things you might not have anticipated.

The way this happens is by shipping a complete product as early as possible. It might not be refined or have every feature implemented, but the core functionality should be present. As soon as this is ready, it should be delivered to the customer (and ideally, the end user). This is where operations comes in.

Oftentimes deployment is left as an after-thought, a last step to check off a list when a project wraps up. This is a mistake. Just as teams take design considerations into account along the way, they should also consider how they will deploy the application.

This doesn’t mean that a team needs to know exactly how their application will be deployed in production for the first iteration, but the tooling to support a production deployment should be put in place as soon as possible. Then, as more is learned about the production environment in each iteration, it can be incorporated into the deployment configuration.

One pattern that that makes a lot of sense for web applications is to use a combination of Puppet or Chef, Vagrant, and Capistrano.

Puppet and Chef

The benefits of using configuration management tools like Puppet and Chef are clear to anyone who’s had to manage a collection of servers over an extended period of time. Documenting server configurations and preventing drift are a couple of the most obvious advantages, but Puppet and Chef also provide a ton of value when it comes to setting up new servers or migrating from one environment (say, AWS) to another (say, your clients’ own internal servers). Which one you use is mostly a matter of preference, but after working with both, Chef seems a little more accessible and faster to set up (though I suspect that Puppet’s declarative DSL may have some long-term benefits for larger or longer-running projects).

Vagrant

Vagrant is extremely popular now, but I still find that I have to make a case for it at times. Many developers working on Macs are used to OS X being UNIX-y enough to port their apps to Linux servers without too much hassle. They often only spin up VMs to work in Microsoft Windows and generally use VMWare Fusion or Parallels for this, not VirtualBox. Why install another virtualization platform and set up another VM?

Automation and Phoenix Servers

Setting up a new VM isn’t all that painful when you have tools like Puppet and Chef (and Vagrant) to automate the process for you. In fact once you have these things in place, you can completely destroy your VM and recreate it with very little effort. This leads to the adoption of a hygienic practice of destroying and rebuilding your environment from code, a pattern Martin Fowler has called the Phoenix Server.

Replicability

By capturing everything essential about your VM’s configuration in code (Puppet/Chef, Vagrant config), you’ve also made this environment easily shareable to the entire team (and beyond). Not only is everyone on the project working with an identical environment (saving countless headaches of “works on my machine”), but it also creates a description of exactly what will be needed in your production environment. Making your configuration manifests and scripts portable across a Vagrant VM and one or more production servers is often only a matter of minor refactoring.

Known Base State

One of the other things that makes Vagrant so useful is the fact that Vagrant VMs are typically created from an existing “base box.” These base boxes are VM images that serve as reusable templates or jumping-off points for individual projects. Generally minimal installs of the most common Linux distros, they are often no more than a few hundred megabytes in size so they don’t take long to download or share, but since they’re reusable, if you have several projects working with the same base box, you’ll only need to install it once. By using generic base boxes, you start with a known state and can share knowledge about that particular platform with others without worrying that you may have made different choices when you first installed the OS because the only differences between your running VM and the common base box will be captured in code. I like the base boxes that Opscode provides along with the Bento project (for creating custom base boxes).

Capistrano

Despite its warts, Capistrano really is a great tool for deploying Ruby on Rails web applications. Capistrano can also serve as an easy way to organize code to be executed either locally or remotely in conjunction with either server configuration tasks or application deployment (separate, but related concerns). Much like Rake within a Ruby app, Capistrano can serve as the glue to tie all of your operational code together.

Summary

These tools make it easy to create a minimum viable product for your first iteration that includes not only application code, but code for managing your infrastructure requirements and deployment needs as well. There is no longer any reason to put off operational concerns until your project is over. Even the smallest web app should be able to deploy to a Vagrant VM with its first iteration. Otherwise, you’re just incrementing.
 

The post Tooling for an Iterative Infrastructure appeared first on Atomic Spin.

BERKS! – Simplifying Chef Solo Cookbook Management with Berkshelf

At Atomic Object, we like to keep things simple. For Justin and I, this often means using Puppet Standalone or Chef Solo to configure servers that don’t yet (and may never) require the coordination of a Puppet Master or Chef Server. Justin recently blogged about the pattern we use with Chef Solo. I’m going to share a new tool we’ve started using to keep these repositories simple, even as the requirements grow.

LEGO My Eggo

As a server configuration repository grows beyond a single monolithic recipe, we refactor it to use several cookbooks like Lego building blocks. Some of these are “weird blocks” specific to the individual needs of a particular server, but some of them are “base blocks” serving a more generic role.

Oftentimes these base blocks have even been implemented already by Opscode or by others in the Chef community. If we re-use the same base blocks over and over for multiple servers, it doesn’t make sense to manually copy those files into each repo every time. There must be a better way.

Ermahgerd! Berkshelf!

Ermahgerd! There is! Cookbooks are Chef’s analog to Ruby’s gems, and Berkshelf lets you treat your cookberks — err… cookbooks the way you treat gems in a modern Ruby project. Just put all of your fravrit — ahem, favorite cookbooks in your Berksfile, add gem "berkshelf" to your Gemfile, bundle install, berks install! It gives me gersberms.

GERSBERMS!

Base Blocks on Your Berkshelf

Seriously though, Berkshelf really does make managing cookbooks easy. Once you’ve installed the berkshelf gem (via your Gemfile with its counterpart, bundler, or with a simple gem install berkshelf), you can run berks init to generate an empty Berksfile (and .gitignore, Gemfile, and Vagrantfile) to get you started.

Adding cookbooks to your Berksfile follows the same pattern as gems in a Gemfile: e.g. cookbook "database", "1.3.6". Whereas with bundler your Gemfile probably has source :rubygems at the top, your Berksfile will have source :opscode.

When you run berks install, Berkshelf installs the cookbooks specified in your Berksfile to your “Berkshelf” usually in ~/.berkshelf/cookbooks/. This works well if you’re working with a Chef Server, but if you’re using Chef Solo the way we do (tarring up all of your server config and scp-ing it to the server using Capistrano), you’ll be happy to know that Berkshelf, like bundler, has an option to “vendorize” the cookbooks in your Berksfile. Simply run berks install --path chef/cookbooks or wherever you’d like to put them.

This works great for Opscode-provided cookbooks, but what about other cookbooks you find on github? No problem — just provide a path to the git repository and Berkshelf will take care of things.

cookbook "mysql", git: "https://github.com/opscode-cookbooks/mysql.git"

You can even specify a branch or tag:

cookbook "mysql", git: "https://github.com/opscode-cookbooks/mysql.git", branch: "1.0.1"

You could also use the new shorthand for repositories hosted on github:

cookbook "artifact", github: "RiotGames/artifact-cookbook", ref: "0.9.8"

Weird Blocks on Your Berkshelf?

All of this makes it really easy to get those base blocks in place. But what about our weird blocks? Do they just get mixed in with all the other ones in chef/cookbooks that Berkshelf is managing for us?

We could leave them in chef/cookbooks, but if we move them somewhere else (say, chef/custom_cookbooks) we can .gitignore chef/cookbooks. We can have Berkshelf copy our weird blocks in by specifying them with a path in our Berksfile like this: cookbook 'radar-console', path: 'chef/custom_cookbooks'.

Adding Berkshelf to Our Chef Solo Pattern

I’ve forked Justin’s repo and added Berkshelf so you can see what it looks like in action.

One thing you’ll notice is that not every cookbook is listed in the Berksfile. That’s because Berkshelf will resolve dependencies specified in a cookbook’s metadata.rb file.

I also added a berks:install task to Capistrano and made sure that it gets executed as the first step of the chef task so that our “vendorized” cookbooks will be copied to the server we’re provisioning. You can see the diff for that here.

Finally, you’ll notice a fair amount of churn in Gemfile.lock. I had to run a bundle update to get Berkshelf to install. It happens to be one of the gems trying to juggle some rather complex dependencies which include the notorious json gem. Be aware that you may also have to bundle update or manually juggle a few dependencies in order to add Berkshelf to your project.

The post BERKS! – Simplifying Chef Solo Cookbook Management with Berkshelf appeared first on Atomic Spin.