Hi! Welcome...

Syndication of blogs and tweets by users of the Freenode ##infra-talk IRC channel

29 February 2012 ~ Comments Off

Better software? or Better Users?

Clarifying my position from this post -

I'm not trying to say people (ops or otherwise) shouldn't want stronger programming skills. I'm saying the equipment we use is pretty shitty.

I am part of the generation raised near devices ever blinking "12:00". Devices which have no business caring what time it is, nor making the device state of "I don't know what time it is" a high priority alert.

It's 2012, and this problem persists - my microwave refuses to cook food unless it has the time *and* date from user input.

I have to learn to program or configure these devices before they'll stop irritating me. And, damn it I hate that.

28 February 2012 ~ Comments Off

Using git cherry to get a list of unmerged and un-cherry-picked commits

Recently I made a post about using Git submodules for Puppet and I pasted some code I use to interact with the repo and ensure I get the workflow right. One of these scripts, promote.sh, had a feature that listed commits not yet in the master branch of each submodule. This was done with git log master..develop. Unfortunately, this command will show commits that have been cherry-picked as not yet merged, since cherry-picks get a new SHA on the branch you bring them into. After doing some digging, I found a new command: git cherry.

Git cherry is made for this exact purpose almost. The man page says it is for determining which commits have not yet made it to upstream, and it does this by comparing the changeset, not the SHA commit id. It will output lines of each commit ID that has not yet been merged to upstream. Then, each line will begin with a - or +. The lines with -’s are already in the branch passed, while the lines beginning with + are the ones that only exist in your current branch. I then took this knowledge, and modified the list function for promote.sh to look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# This will go through all submodules and run the appropriate git command to
# show what commits master needs to be on track with develop. This uses the
# git cherry command, which is basically built for this exact purpose. Note
# that this had to be used due to our use case of sometimes promoting single
# changes through cherry-pick. Therefore git log master..develop would some-
# times show inaccurate information.
function list_pending_promotions() {
   pushd $puppetroot >/dev/null 2>&1

   read -p "Update checkout? (y/N) " answer

   if [ "$answer" != "y" -a "$answer" != "Y" ]; then
      printf "Be warned, data might be innaccurate if you are not up to date.\n"
   else
      printf "Ensure checkout is updated...\n"
      scripts/updatecheckout.sh
   fi

   printf "Show pending changes...\n\n"
   printf "========================================\n"
   git submodule --quiet foreach 'if [[ $path =~ ^production.* ]]; then printf "%s:\n\n" "$(basename $name)"; changes=$(git checkout develop 2>/dev/null; git cherry -v master | egrep "^\+" | sed "s/^+ //"; git checkout master 2>/dev/null); if [ -z "$changes" ]; then printf "No pending commits.\n"; else printf "%s\n" "$changes"; fi; printf "========================================\n"; fi'
   popd $puppetroot >/dev/null 2>&1
}

The long line for git submodule is kind of suboptimal, but I didn’t really want to play with escaping at this point. This feature is nice though and after asking around not many people seemed to know about it.

28 February 2012 ~ Comments Off

Hackery to script creation of new Puppet CA

One of the tasks when setting up a new master is to set up a new CA. This is not something most people think about because Puppet does it automatically for you. However, unless your puppet.conf is in place already, it may not create 4096 bit keys. Also, if you’re doing a completely automated setup of a new organization, you might want to just run a script to generate the CA in a version controlled directory.

So, here is my quick and dirty hack for generating a new Puppet CA:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env ruby
# QnD for generating Puppet CA. Doing a certification chain is a royal PITA
# with Puppet, so I typically have the Puppet CA as a separate entity from
# the company SSL setup. This is obviously suboptimal for enterprise-scale
# Puppet setups. You've been warned!

require 'rubygems'
require 'puppet'

begin
   Puppet::SSL::Host.ca_location = :local
   Puppet.settings.use :ca
   Puppet[:certname] = "puppet"
   Puppet[:keylength] = "4096"
   Puppet[:ssldir] = "./"
   newca = Puppet::SSL::CertificateAuthority.new
   newca.generate_ca_certificate
   Puppet::SSL::CertificateAuthority.instance
rescue
   STDERR.puts "Error creating Puppet CA!!!"
   exit -2
end

27 February 2012 ~ Comments Off

Sensu handler sets

In past articles we have covered some of basics of Sensu handlers. A nice feature we haven't touched on yet is handler "sets". Handler sets were added around v0.9.2 and can be quite useful for saving time when modifying your handler.

For example, consider you have a standard set of handlers that you assign to most of your checks — pagerduty, irc, campfire. Now, suppose you want to add the GELF (graylog2) handler to all of your monitors as well. If each of your checks is defined as such:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["pagerduty", "irc", "campfire"]
    }
  }
}

… you would need to modify every check's "handlers" attribute to include your new "gelf" handler. If you have a lot of checks this can be a little bit of a burden.

Handler sets save a lot of time in this situation. If you instead define your checks like this:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["default"]
    }
  }
}

And then define your handlers on your Sensu server like the following. Notice we have converted the 'default' handler to a set.

"handlers": {
      "default": {
        "type": "set",
        "handlers": ["pagerduty", "irc", "campfire", "gelf"]
      },
      "pagerduty": {
        "type": "pipe", 
        "command": "/etc/sensu/handlers/pagerduty"
      },
      "irc": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/irc"
      },
      "campfire": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/campfire"
      },
      "gelf": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/gelf.rb"
      }   
    },

You only need to modify the "default" handler set to add the new "gelf" handler and no need to change any of your check definitions.

It's a simple feature, but it can save a bunch of time. You could also setup a handler set specifically for metrics and another set for notifications. In the case of metrics, you could easily ship metrics to multiple systems – Graphite, Librato, Cube, etc. And adding a new system would be as simple as creating the handler and adding it to the handler set. Happy Sensu'ing.

27 February 2012 ~ Comments Off

Beyond Bundler: A Configuration Management Starter Kit

Configuration management or “infrastructure as code” can provide a common language for application developers and operations specialists alike to describe the infrastructure requirements of an application. By capturing these requirements in code, bootstrapping becomes a repeatable process, and insights from operations teams supporting the application in a production environment can be fed back to the developers in a virtuous cycle.

As an example of what this might look like with some current tools, I’ve created a starter kit for using vagrant, veewee, and a bit of puppet to automate the building of virtualized infrastructure for a Rails 3 application. The end result is a VirtualBox virtual machine described in code (from a Veewee basebox definition of the basic virtual hardware to a Puppet manifests describing the necessary packages and bootstrapping). This means that down the road, an environment in which your application will run can be repeatedly built and all of the steps of that process are both visible and modifiable, with changes captured in source control.

Find it on GitHub, here.

There are a few things I haven’t finished wiring together as of this writing, but it should be enough to see how the main pieces fit together.

The project makes as few assumptions as possible about your environment. It assumes that you have a recent version of VirtualBox installed, RVM installed, and ruby-1.9.3-p125 installed via RVM – from there the project rvmrc and bundler should take care of the remaining dependencies.

To run it (build a new VM from scratch and deploy your app to it), you’ll want to run the following commands:
vagrant basebox build demo-centos-box
vagrant basebox validate demo-centos-box
vagrant basebox export demo-centos-box
vagrant up
cap environment:vagrant deploy:setup
cap environment:vagrant deploy

As of this writing, I haven’t added the hooks to actually launch the application server, but you can start WEBBrick by hand like so:
vagrant ssh
…and then from within the VM:
cd sites/demo-app/current && RAILS_ENV=production bundle exec rails s

Then, in your browser visit 33.33.33.10:3000. Tada!

This was a weekend project, and it’s likely that I’ve overlooked some things, but I plan to continue honing it. Let me know in the comments (or in pull requests) what’s still broken. One of the major motivations of this approach is getting past the issue of “works for me” – so if it doesn’t work for you, I want to know! Thanks.

27 February 2012 ~ Comments Off

One Liner: Ifstat

Recently I tried ifstat on a freshly updated linux box with a 3.2 kernel, it was reporting nothing (0kb) although the network was heavily used.

A quick one-liner confirmed this and I decided to keep it here for later use:

export IFACE=eth0 ; while(:); do cat /proc/net/dev;sleep 1;done|nawk -W interactive -v iface=${IFACE} 'BEGIN{ cnt = 0 } $1 ~ iface { if (cnt) { print iface ":",(($2 - cnt)/1024),"Kbytes/sec"; } cnt=$2; }'

26 February 2012 ~ Comments Off

My technique for reconciling generic canned Puppet modules and organization-specific configurations

As I’ve mentioned before I use a large set of cross-platform, relatively generic modules that I wrote for a system I call GRUMPS. The problem with this method is that you’re going to need to have some ERBs and files that are custom to your organization.

In my previous post about using Puppet submodules, you may have noticed that in addition to my grumps-modules, I use an organization-modules as the second module in my modulepath. So, to remediate the need for site- specific stuff while maintaining the generic moduleset is to create modules under the organization-modules directory named modulename_priv.

Then I do something like this to tell my defined resources to look there (the …’s cut out code not necessary for this demonstration):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
define httpd::site (
   $ensure  = "present",
   $group   = "${httpd::apache_group}",
   $mode    = "640",
   $owner   = "root",
   $private = "false"
) {
...

   if ($private == "true") {
      $template_path = "httpd_priv/sites/${name}.erb"
   } else {
      $template_path = "httpd/sites/${name}.erb"
   }

...

   file {
      "${httpd::confdir}/${name}":
         content  => $ensure ? { present => template($template_path), absent => undef },
         ensure   => $ensure,
         group    => $group,
         mode     => $mode,
         owner    => $owner,
         tag      => "httpd_config";
   }

26 February 2012 ~ Comments Off

Managing email forwarding in Exim with Puppet

I have a number of mail servers where mail enters, get spam scanned etc and then forwarded to mail box servers. This used to be customer facing and had web interfaces and statistics etc but I am now scaling all this down to just manage my own and some friends domains.

Rather than maintain all the web interfaces that I really could not care for I’d rather manage this with Puppet, my ideal end result would be:

exim::route{"devco.net":
  nexthop         => "my.mailbox.server",
  spamthreshold   => 10,
  spamdestination => ":blackhole:",
  has_greylist    => 1,
  has_spam_check  => 1,
  has_whitelist   => 1
}

This should add all the required configuration to deliver mail arriving at the mail relay for devco.net to the server my.mailbox.server. It will set up Spam Assassin scans and send all mail that scores more than 10 to the exim specific destination :blackhole: that would simply delete the mail. I could specify any valid mail destination here like a file or other email address. I won’t be covering the has_* entries in this guide, they just control various policies in my ACLs on a per domain basis.

I’ll first cover the Exim side of things, clearly I do not want to be editing exim.conf each time so I will read the domain information from a file stored on the server. These files will be stored in /etc/exim/routes/devco.net and look like:

nexthop: my.mailbox.server
spamthreshold: 10
spamdestination: :blackhole:

In order to accept mail for a domain Exim needs a list of valid domains it will accept mail for, so as our routes are named after the domain we can just leverage that to build the list:

domainlist mw_domains = dsearch;/etc/exim/routes

Next we should pull from the file the various settings we store there:

NEXTHOP = ${lookup{nexthop}lsearch{/etc/exim/routes/${domain}}}
DOMAINREJECTSCORE = ${eval:10*${lookup{spamthreshold}lsearch{/etc/exim/routes/${domain}}}}
DOMAINSPAMDEST = ${lookup{spamdestination}lsearch{/etc/exim/routes/${domain}}}
 
ACL_SPAMSCORE = acl_m3

This creates handy variables that we can just use in our routes and spam configuration, I won’t go into the actual setup of spam assassin scanning as that’s pretty standard stuff better documented elsewhere. In the spam assassin ACLs just store your $spam_score_int in ACL_SPAMSCORE.

To deliver the mail either to the specific spam destination or to the next hop we just need to add 2 routers to the routes section. These are order dependant so they should be in the order below:

spamblock:
  driver          = redirect
  condition       = ${if >= {$ACL_SPAMSCORE}{DOMAINREJECTSCORE}{true}{false}}
  data            = DOMAINSPAMDEST
  headers_add     = X-MW-Note: Redirecting mail to domain spam destination
  domains         = +mw_domains
  no_verify

Here we’re just doing a quick if check over the stored spam score to see if its bigger or equal to the threshold stored in DOMAINREJECTSCORE and then set the data of the route – where the mail should go – to the configured address from DOMAINSPAMDEST. This router will only be active for domains that this Exim server is a relay for and it adds a little debug note as a header.

The actual mail delivery that is being used in place of the normal dnslookup route is here:

mw_domains:
  driver          = manualroute
  transport       = remote_smtp
  domains         = +mw_domains
  user            = root
  headers_add     = "X-MW-Recipient: ${local_part}@${domain}\n\
                     X-MW-Sender: $sender_address\n\
                     X-MW-Server: $primary_hostname"
  route_data      = MW_NEXTHOP

This router is also restricted to only our relay domains, it adds some headers for debug purposes and finally sets the route_data of the email to the next hop from MW_NEXTHOP thus delivering the mail to the destination.

That’s all there is to do on the Exim side, it’s pretty standard stuff. Next up the Puppet define:

define exim::route($nexthop, $spamthreshold, $spamdestination, $ensure = "present") {
  file{"/etc/exim/routes/${name}":
    ensure  => $ensure,
    content => template("exim/route.erb")
  }
}

And the template for this define is also extremely simple:

nexthop: <%= nexthop %>
spamthreshold: <%= spamthreshold %>
spamdestination: <%= spamdestination %>

I could stop here and just create a bunch of exim::route resources but that would be code changes, I prefer just changing data. So I am going to create a JSON file called mailrelay.json and store it with my Hiera data.

{
  "relay_domains": {
    "devco.net": {
      "nexthop": "my.mailbox.server",
      "spamdestination": ":blackhole:",
      "spamthreshold": 10,
      "has_dkim": 1
    },
    "another.com": {
      "nexthop": "ASPMX.L.GOOGLE.COM.",
      "spamdestination": ":blackhole:",
      "spamthreshold": 10
    }
  }
}

I assign all my incoming mail servers a single class that would look roughly like this:

class roles::mailrelay {
  include exim
  include exim::mailrelay
 
  $routes = hiera("relay_domains", "", "mailrelay")
  $domains = keys($routes)
 
  exim::routemap{$domains:
    routes => $routes
  }
}

The call to Hiera fetches the entire hash from the mailrelay.json file and stores it in $routes. I then use the keys function from puppetlabs-stdlib to extract just the list of domains into an array. I then pass that into a define exim::routemap that iterates the list and builds up individual exim::route resources.

The routemap define is just as below, I’ve shortened it a fair bit as I also have validation logic in here to make sure I pass valid data in the hash from Hiera, the stdlib module has various validator functions thats really handy for this:

define exim::routemap($routes) {
  exim::route{$name:
    nexthop => $routes[$name]["nexthop"],
    spamthreshold => $routes[$name]["spamthreshold"],
    spamdestination => $routes[$name]["spamdestination"]
  }
 
  if ($routes[$name]["has_dkim"] == 1) {
    exim::dkim_domain{$name: }
  } else {
    exim::dkim_domain{$name: ensure => absent}
  }
}

And that’s about it, now my mail routing setup, DKIM signing and other policies are managed in a simple JSON file in my Puppet Manifests.

26 February 2012 ~ Comments Off

Xcode Command Line Tools

Recently, Apple did the most awesome thing for non-Xcode developers.

They made the development tools available as a standalone package!

This is awesome news for those of us who only haveWhad Xcode installed to install RubyGems that compile native extensions, or for installing software with Homebrew, MacPorts or similar.. You can download them by logging into the Developer Download site.

This appears to be work started by Kenneth Reitz with his OSX GCC Installer project. I did try that project out, but ran into issues I didn’t resolve right away, so I reverted to using Xcode proper. However with the package from Apple I don’t seem to have any issues so far.

If you already have Xcode installed, you may want to remove it first.

1
2
sudo /Developer-3.2.6/Library/uninstall-devtools
sudo /Developer/Library/uninstall-devtools –mode=all /thx

My understanding is that you can remove the /Developer* director(y|ies) when complete. I had ollllld Xcode on the system where I first did this.

Next, download and install the package from Apple. It’s about 170M and takes only a couple minutes to install; sorry I don’t have a Chef recipe for this ;).

I did run into an issue with Homebrew where it wasn’t finding the right gcc binary. I had to run the following commands to fix that issue.

1
2
sudo xcode-select -switch /usr/bin
sudo ln -sf /usr/bin/llvm-gcc-4.2 /usr/bin/gcc-4.2

You wouldn’t think that something like an 8G installation would matter in 2012. However, disk space is a precious commodity on MacBook Airs and systems that have SSDs as the root volume. This is very welcome change for me, especially since it means that future Mac OS X installations do not require a large download before I can start doing things that get my system ready to use.

26 February 2012 ~ Comments Off

Recipe for Building Emacs

It is no secret that I use GNU Emacs as my default text editor. It is perhaps less evident but no less relevant that I use Emacs 24. I really like the built-in color theme support and the package management system for getting the various modes I like to use.

Recently, I revamped my Emacs configuration. This post isn’t about that topic. Instead, this post is about how I made sure that all the systems I want to use Emacs on have the latest version available.

Unfortunately, Emacs 24 is still unreleased, so it is not available as the default package on the distributions I use for my personal systems (Ubuntu/Debian flavors). I wrote a recipe to install Emacs from source. This is easy enough to do, but since I already automate everything on my home network with Chef, it was a natural fit for a new recipe. I simply added this to my local emacs cookbook, in recipes/source24.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
srcdir = "#{Chef::Config[:file_cache_path]}/emacs-source"

%w{ git-core build-essential texinfo autoconf libncurses-dev }.each {|prereq| package prereq}

git "#{Chef::Config[:file_cache_path]}/emacs-source" do
  repository "git://git.savannah.gnu.org/emacs.git"
  action :checkout
end

bash "build emacs24" do
  cwd srcdir
  creates "#{srcdir}/src/emacs"
  code <<-EOH
    ./autogen.sh && \
    ./configure –without-x && \
    make bootstrap && \
    make 2>&1 >| make-#{node.name}-#{node['ohai_time']}
  EOH
end

execute "install emacs24" do
  cwd srcdir
  command "make install 2>&1 >| make-#{node.name}-#{node['ohai_time']}"
  creates "/usr/local/bin/emacs"
  only_if "#{srcdir}/src/emacs –version"
end

Then I updated my base role to replace recipe[emacs] with recipe[emacs::source24] and ran Chef. It took about 25 minutes to do the build, but now I have the same version of Emacs everywhere, and there was much rejoicing.

And yes, you’re absolutely right, I could just build a package and install that. However, I don’t want to set up and maintain a package management repository for my small network, as easy as that may be.

My OS X systems are a special case because I’m using Homebrew, but the homebrew cookbook does not [yet?] support install-time options, and I didn’t spend the time adding support for building the OS X Emacs w/ cocoa support from git. When I tackle that, I’ll make another post, so stay tuned!