These two improvements come under the banner of 'External Facts'. The first allows you to surface your own facts from a static file, either plain text key value pairs or a specific YAML / JSON format. These static files should be placed under /etc/facter/facts.d

$ sudo mkdir -p /etc/facter/facts.d

# note - the .txt file extension
$ echo 'external_fact=yes' | sudo tee /etc/facter/facts.d/external_test.txt
external_fact=worked

$ facter external_fact
worked

At its simplest this is a way to surface basic, static, details from system provisioning and other similar large events but it's also an easy way to include details from other daemon and cronjobs. One of my first use cases for this was to create 'last_backup_time' and 'last_backup_status' facts that are written at the conclusion of my backup cronjob. Having the values inserted from out of band is a much nicer prospect that writing a custom fact that parses the cron logs.

If that's a little too static for you then the second usage might be what you're looking for. Any executable scripts dropped in the same directory that produce the same output formats as allowed above will be executed by facter when it's invoked.

# scripts must be executable!
$ sudo chmod a+rx /etc/facter/facts.d/process_count

$ cat /etc/facter/facts.d/process_count
#!/bin/bash

count=$(ps -efwww | wc -l | tr -s ' ')
echo "process_count=$count"

$ facter process_count
209

The ability to run scripts that provide facts and values makes customisation easier in situations where ruby isn't the best language for the job. It's also a nice way to reuse existing tools or for including information from further afield - such as the current binary log in use by MySQL or Postgres or the hosts current state in the load balancer.

While there have been third party extensions that provided this functionality for a while it's great to see these enhancements get included in core facter.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!]]> http://infra-talk.org/2013/05/15/facter-1-7-and-external-facts/feed/ 0 http://feedproxy.google.com/~r/AuxesisMusings/~3/Ujdt8FOpJ8Q/ http://feedproxy.google.com/~r/AuxesisMusings/~3/Ujdt8FOpJ8Q/#comments Tue, 14 May 2013 14:00:00 +0000 Lindsay Holmwood http://holmwood.id.au/~lindsay/2013/05/15/escalating-complexity-af447 Back in 2009 when I was backpacking around Europe I remember waking up on the morning of June 1 and reading about how an Air France flight had disappeared somewhere over the Atlantic.

The lack of information on what happened to the flight intrigued me, and given the traveling I was doing, I was left wondering "what if I was on that plane?"

Keeping an ear out for updates, in December 2011 I stumbled upon the Popular Mechanics article describing the final moments of the flight. I was left fascinated by how a technical system so advanced could fail so horribly, apparently because of the faulty meatware operating it.

Around the same time I began reading the works of Sidney Dekker. I was left in a state of cognitive dissonance, trying to reconcile the mainstream explanation of what happened in the final moments of AF447 (the pilots were poorly trained, inexperienced, and simply incompetent) with the New View that the operators were merely locally rational actors within a complex system, and that "root cause is simply the place you stop looking further" - with that cause far too commonly attributed to humans.

I decided to do my own research, which resulted in me producing a talk that has received the strongest reaction of any talk I've ever given.

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems.

Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand.

While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated.

In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

The subject matter is heavy, and I while it's something I'm passionate about, it was an emotionally taxing talk to prepare, and a talk that angers me when presenting.

Time to let it sit and rest.

Out of this track emerged some interesting debate around the viability of the whole notion of a cloud futures market. Jonathan Murray, EVP & Chief Technology Officer at Warner Music Group, followed up with an article presenting his perspective, from the skeptical end of the spectrum. This article presents a view from the optimistic side.

I’m indebted to John Woodley, investor and board advisor at Strategic Blue, for his insights and comments. John is a former Managing Director at Morgan Stanley, where he was co-Head of non-oil commodities for EMEA. He was the first person to be hired by a Wall Street bank from a power utility, becoming a founder member of the Morgan Stanley electricity trading desk. As one of the world’s foremost authorities on commodity market development and the pricing of non-standard pseudo-commodity products, his perspective is fascinating.

The challenge

Jonathan explicitly self-identifies as one who is unconvinced about the prospect of a futures market for cloud. His article is fairly long, and his argument well made. The core point he puts forward is essentially: “There is no volatility or friction so we don’t need intermediation.”

Is there friction in the cloud market?

Jonathan doesn’t explicitly define friction, but I think he means that in today’s world, primary buyers can easily find primary sellers, and as such there’s no need for an intermediary. This is a fair point. In the old world, this was impossible. In a market without intermediaries, in order to get the best price, you need each of the buyers to make as many calls as are there are sellers. Let the number of buyers in the market be ‘b’ and the number of sellers be ‘s’. In a tiny market, with 10 buyers and 10 sellers, you need b*s calls in order to achieve the best price, or 100 calls in total.

However, in a world in which buyers are also resellers, the number of calls needed in order to secure the best price becomes much larger - somewhere around the factorial of the number of buyers plus the number of sellers - (b+s)!. In our tiny market, (10+10)! is 2432902008176640000. If one entity in the middle takes calls and keeps note you only need b+s calls and not (b+s) factorial. This is a conventional argument for the existence of a market - it offers huge savings. On these grounds, it’s hard to imagine Jonathan thinks there isn’t a market, a forum for price comparison, but certainly in today’s world the web is that intermediary near as dammit is to swearing. Jonathan correctly describes markets as places where supply is matched to demand, and indeed, we use markets because they have proven better than manual or -so far- computational planners in correctly matching supply and demand efficiently.

There is, of course, very significant friction in the cloud market, per a different definition. Look at the way cloud providers sell their services. They want long-term commitment, in their own currency, by credit card. By contrast, the cloud user wants minimal commitment, the best flexibility to move should a better deal emerge, or a better technology offering appear. They’d rather not have to keep a credit card on file, and in many cases, for example in India, currency fluctuations make a huge impact, and they’d greatly prefer to pay in their own local currency. In this situation, an intermediary who buys from the providers on terms which suit the provider, and sells to the buyers on terms which suit the buyers is very attractive.

Is there a need for intermediation?

Let’s examine whether there’s a need for intermediation against the assertion that there is no volatility. I think that assertion is false. It strikes me as similar to the assertion that there was no petrol price volatility in the US in 1979 or more recently in the Northeastern US after Hurricane Sandy, because the posted price at the pump did not change. The reality of course is that the underlying value changed so radically that people would queue for hours to get the scarce commodity and even threaten each other with violence to get precedence.

To be sure we have not seen that in cloud yet but we have seen scarcities. John Woodley cited an exemplary case with which he was particularly familiar. The Morgan Stanley IT department was approached to lend space at a datacenter because a major movie producer could not get enough capability on the open market to render a movie before the all-important Christmas release date. Had the IT department called a Fixed Income trader before agreeing to the deal I am fairly sure the producer would have had to pay more in cash or kind than a few T Shirts for the deal.

So, I have to ask: Is there no state of the world in which a sudden demand for many thousands or millions of instances might occur? Are there that many sitting idle? No, a far more productive and interesting line of thought is how the demand surge might occur and how to be in the right place at the right time to benefit. For an interesting example, consider the events surrounding Silver Thursday, in 1980, in which the Hunt Brothers had hoarded a third of the world’s supply of silver, inflating the price by a whopping 700%. At the same time, anecdotally, a certain bank held for some strange reason a large part of the silver smelting capability, and thus were able to charge greatly increased prices for people wanting to melt down the family silver, to sell as ingots. In John Woodley’s view, it is tremendously likely that we will see scarcity and volatility in the cloud markets, and a wealth of opportunity and demand to intermediate.

What about contractual preference?

I touched on this when discussing the idea of friction in the cloud market. A second but no less important driver for intermediation was not addressed in Jonathon’s article. In certain markets, the contractual preferences of buyers and sellers diverge dramatically. In such cases the insertion of a financial intermediary is very helpful and both parties are prepared to pay the rather minimal cost such intermediation causes. An example that is blindingly obvious can be found in electricity. Power plants are only used when the demand is sufficiently high to cover the operating expenses of the power plant. Power plants are built on borrowed money. A power plant owner needs to finance the debts used to build the plant, and so wants a fixed payment per month from their customers. By contrast, a retailer of electricity simply wants to be charged on a per use basis, and wants nothing to do with fuel cost pass-through, or a fixed monthly fee. Think about the way datacenters are funded. It’s exactly the same model - why do you think cloud providers offer such a huge discount for a commitment to usage? It’s the same! In the electricity markets, intermediation provides a welcome service, in which the intermediary is the buyer to every seller and the seller to every buyer, ironing out the contractual wrinkles, and benefiting the market as a whole. There’s already evidence of demand for this kind of service in the cloud market today.

Summary

The fundamental point Jonathan makes in his argument is that because there is no volatility or friction, there’s no need for intermediation. However, there is already evidence of price fluctuation and scarcity, and in terms of the delta between the way buyers want to buy and sellers want to sell, there is clearly friction. The cloud market behaves like and looks like a blend of the electricity and coal market, both of which are highly intermediated and heavily traded. Yes, it’s early days, but it seems very likely indeed that we’ll see the same kind of behavior in the cloud.

Well, that time has come. I wanted to deploy a later version of capistrano across a client’s infrastructure and my previous approach didn’t work so I grabbed fpm and did this:

mkdir ~/tmp/gems
cd ~/tmp/gems
gem install --no-ri --no-rdoc --install-dir . capistrano
find ./cache -name '*.gem' | xargs -rn1 fpm -s gem -t rpm
ls *.rpm
rubygem-capistrano-2.15.4-1.noarch.rpm	rubygem-net-scp-1.1.0-1.noarch.rpm   rubygem-net-ssh-2.6.7-1.noarch.rpm
rubygem-highline-1.6.19-1.noarch.rpm	rubygem-net-sftp-2.1.2-1.noarch.rpm  rubygem-net-ssh-gateway-1.2.0-1.noarch.rpm

Nice and easy. Kudos whack!

First off, the whole thing is in this gist. I’m going to break it down into sections below. The file is spec/default_spec.rb in the java cookbook (not committed/pushed yet).

The chefspec gem is where all the magic comes from. You can read about ChefSpec on its home page. You’ll need to install the gem, and from there, run rspec to run the tests.

1

require 'chefspec'

Next, we’re going to describe the default recipe. We’re using the regular rspec “let” block to set up the runner to converge the recipe. Then, because we know/assume that the openjdk recipe is the default, we can say that this chef run should include the java::openjdk recipe.

1
2
3
4
5

describe 'java::default' do
  let (:chef_run) { ChefSpec::ChefRunner.new.converge('java::default') }
  it 'should include the openjdk recipe by default' do
    chef_run.should include_recipe 'java::openjdk'
  end

Next, this cookbook supports Windows. However, we have to set up the runner with the correct platform and version (this comes from fauxhai), and then set attributes that are required for it to work.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

context 'windows' do
    let(:chef_run) do
      runner = ChefSpec::ChefRunner.new(
        'platform' => 'windows',
        'version' => '2008R2'
        )
      runner.node.set['java']['install_flavor'] = 'windows'
      runner.node.set['java']['windows']['url'] = 'http://example.com/windows-java.msi'
      runner.converge('java::default')
    end
    it 'should include the windows recipe' do
      chef_run.should include_recipe 'java::windows'
    end
  end

Next are the contexts for other install flavors. The default recipe will include the right recipe based on the flavor, which is set by an attribute. So we set up an rspec context for each recipe, then set the install flavor attribute, and test that the right recipe was included.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

  context 'oracle' do
    let(:chef_run) do
      runner = ChefSpec::ChefRunner.new
      runner.node.set['java']['install_flavor'] = 'oracle'
      runner.converge('java::default')
    end
    it 'should include the oracle recipe' do
      chef_run.should include_recipe 'java::oracle'
    end
  end
  context 'oracle_i386' do
    let(:chef_run) do
      runner = ChefSpec::ChefRunner.new
      runner.node.set['java']['install_flavor'] = 'oracle_i386'
      runner.converge('java::default')
    end
    it 'should include the oracle_i386 recipe' do
      chef_run.should include_recipe 'java::oracle_i386'
    end
  end

Finally, a recent addition to this cookbook is support for IBM’s Java. In addition to setting the install flavor, we must set the URL where the IBM Java package is (see the README in the commit linked in that ticket for detail), and we can see that the ibm recipe is in fact included.

1
2
3
4
5
6
7
8
9
10
11
12

  context 'ibm' do
    let(:chef_run) do
      runner = ChefSpec::ChefRunner.new
      runner.node.set['java']['install_flavor'] = 'ibm'
      runner.node.set['java']['ibm']['url'] = 'http://example.com/ibm-java.bin'
      runner.converge('java::default')
    end
    it 'should include the ibm recipe' do
      chef_run.should include_recipe 'java::ibm'
    end
  end
end

This is just the start of the testing for this cookbook. We’ll need to test each individual recipe. However as I’ve not written that code yet, I don’t have examples. Stay tuned!

This is all powered by a relatively simple cookbook, and some click-click-clicking in the Jenkins UI. I’ll walk through what I did to set up my Jenkins system.

First, I started with Debian 7.0 (stable, released this past weekend). I installed the OS on it, and then bootstrapped with Chef. The initial test was to make sure everything installed correctly, and the commands were functioning. This was done in a VM, and is now handled by test-kitchen itself (how meta!) in the cookbook, kitchen-jenkins.

The cookbook, kitchen-jenkins is available on the Chef Community site. I started with a recipe, but extracted it to a cookbook to make it easier to share with you all. This is essentially a site cookbook that I use to customize my Jenkins installation so I can run test-kitchen builds.

I apply the recipe with a role, because I love the roles primitive in Chef :-). Here is the role I’m using:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

{
  "name": "jenkins",
  "description": "Jenkins Build Server",
  "run_list": [
    "recipe[kitchen-jenkins]"
  ],
  "default_attributes": {
    "jenkins": {
      "server": {
        "home": "/var/lib/jenkins",
        "plugins": ["git-client", "git"],
        "version": "1.511",
        "war_checksum": "7e676062231f6b80b60e53dc982eb89c36759bdd2da7f82ad8b35a002a36da9a"
      }
    }
  },
  "json_class": "Chef::Role",
  "chef_type": "role"
}

The run list is only slightly different here than my actual role, I have a few other things in the run list, which are other site-specific recipes. Don’t worry about those now. The jenkins attributes are set to ensure the right plugins I need are available, and the right version of jenkins is installed.

(I’m going to leave out the details such as uploading cookbooks and roles, if you’re interested in test-kitchen, I’ll assume you’ve got that covered :-).)

Once Chef completes on the Jenkins node, I can reach the Jenkins UI, conveniently enough, via “http://jenkins:8080” (because I’ve made a DNS entry, of course). The next release of the Jenkins cookbook will have a resource for managing jobs, but for now I’m just going to create them in the webui.

For this example, I want to have two kinds of cookbook testing jobs. The first, is to simply run foodcritic and fail on any correctness matches. Second, I want to actually run test-kitchen.

A foodcritic job is simple:

1. New job -> Build a free-style software project “foodcritic-COOKBOOK”.
2. Source Code Management -> Git, supply the repository and the master branch.
3. Set a build trigger to Poll SCM every 5 minutes, once an hour, whenever you like.
4. Add a build step to execute a shell, “foodcritic . -f correctness”

I created a view for foodcritic jobs, and added them all to the view for easy organizing.

Next, I create a test-kitchen job:

1. New job -> Copy existing job “foodcritic-COOKBOOK”, name the new job “test-COOKBOOK”.
2. Uncheck Poll SCM, check “Build after other projects are built” and enter “foodcritic-COOKBOOK”.
3. Replace the foodcritic command in the build shell command with “kitchen test”.

Now, the test kitchen test will only run if the foodcritic build succeeds. If the cookbook has any correctness lint errors, then the foodcritic build fails, and the kitchen build won’t run. This will help conserve resources.

Hopefully the kitchen-jenkins cookbook is helpful and this blog post will give you some ideas how to go about adding cookbook tests to your CI system, even if it’s not Jenkins.

First, the issue manifests itself only on Debian systems. The runit cookbook’s runit_service provider will write an LSB init.d script on Debian, rather than symlinking to /usr/bin/sv. The problem raised in the new ticket is that the template will follow the link and write to /usr/bin/sv. This is bad, as it will end up in a forkbomb as runsvdir attempts to restart sv on all the things. Oops! Sorry about that. Let’s get it fixed, and practice some TDD.

The runit cookbook includes support for test-kitchen, though I did need to update it for this effort. Part of this change was adding a box for Debian in the .kitchen.yml. I set about resolving this with TDD in mind.

First, the runit cookbook includes a couple “test” cookbooks to facilitate setting up the system with the runit_service resource so the outcome can be tested to ensure the behavior is correct. I started by adding a “failing test” in the runit_test::service recipe, meaning a link resource, and a runit_service resource that would overwrite /usr/bin/sv.

1
2
3
4
5
6
7

link "/etc/init.d/cook-2867" do
  to "/usr/bin/sv"
end

runit_service "cook-2867" do
  default_logger true
end

Then I ran kitchen test on the Debian box. As expected, the link was created, and then the runit service was configured. The service’s provider will wait until the service is up. Since we’ve destroyed the sv binary, that will never happen, so I destroyed it. I manually confirmed the behavior too, to make sure I wasn’t seeing something weird. Due to its very nature, this is really hard to test for automatically, but it will happen consistently.

Next, I had to write the code to implement the fix for this bug. Essentially, this means checking if the /etc/init.d/cook-2867 file is a symbolink link, and removing it.

1
2

initfile = ::File.join( '/etc', 'init.d', new_resource.service_name)
::File.unlink(initfile) if ::File.symlink?(initfile)

Simple enough. Next I tested again by destroying the existing environment and rerunning it from scratch. This takes some time, but it verifies that everything is working properly. Here’s the output on Debian:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

INFO: Processing link[/etc/init.d/cook-2867] action create (runit_test::service line 147)
INFO: link[/etc/init.d/cook-2867] created
INFO: Processing service[cook-2867] action nothing (dynamically defined)
INFO: Processing runit_service[cook-2867] action enable (runit_test::service line 151)
INFO: Processing directory[/etc/sv/cook-2867] action create (dynamically defined)
INFO: Processing template[/etc/sv/cook-2867/run] action create (dynamically defined)
INFO: Processing directory[/etc/sv/cook-2867/log] action create (dynamically defined)
INFO: Processing directory[/etc/sv/cook-2867/log/main] action create (dynamically defined)
INFO: Processing directory[/var/log/cook-2867] action create (dynamically defined)
INFO: Processing file[/etc/sv/cook-2867/log/run] action create (dynamically defined)
INFO: Processing template[/etc/init.d/cook-2867] action create (dynamically defined)
INFO: template[/etc/init.d/cook-2867] updated content
INFO: template[/etc/init.d/cook-2867] owner changed to 0
INFO: template[/etc/init.d/cook-2867] group changed to 0
INFO: template[/etc/init.d/cook-2867] mode changed to 755
INFO: runit_service[cook-2867] configured
INFO: Chef Run complete in 7.267132764 seconds
INFO: Running report handlers

I didn’t feel I needed a specific test for this in minitest-chef, because it wouldn’t have finished converging (earlier behavior I saw in the “failing” test).

If you’re contributing to cookbooks, and they have support for test-kitchen, it’s awesome if you can open a bug report with a failing test. In this case, it was fairly easy to reproduce the bug.

Below you'll find a simple metaparameter (a parameter that works with any resource type) that adds this feature to puppet. As this is an early prototype I've hacked it directly in to my local puppet fork. Below you'll see a sample resource that declares a deprecation date and message, the code that implements it and a simple command line test you can run to confirm it works.

# sample puppet resource using :deprecation

  file { '/ec/cron.d/remove_foos':
    ensure      => 'file',
    source      => 'puppet:///modules/foo/foo.cron',
    deprecation => '20130425:Release 6 removes the need for the foo cronjob',
  }


  $ sudo vi puppet-3.1.1/lib/puppet/type.rb

  newmetaparam(:deprecation) do
    desc "
      Add a deprecation warning to resources.

      file { '/etc/foo':
        content     => 'Bar',
        deprecation => '20130425:We no longer need the foo'
      }

      The deprecation comes in two parts, separated by a :
      The date is in format YYYYMMDD and the message is a free form string.
    "

      munge do |deprecation|
        date, message = deprecation.split(':')

        # YYY MM DD - one true timestamp
        now = Time.now.strftime('%Y%m%d')

        if (now >= date)
          rsrc = "#{@resource.type.capitalize}[#{@resource.name}]"

          Puppet.warning "#{rsrc} expired on #{date}: #{message}"
        end
      end
    end

# command line test


$ puppet apply -e 'file { "/tmp/dep": content => "foo\n", deprecation =>
"20120425:We can remove this file after release 4" }' 
Warning: File[/tmp/dep] expired on 20120425: We can remove this file after release 4
Notice: Finished catalog run in 0.06 seconds

Using the metaparameter is easy enough, just specify 'deprecation' as a property on a resource and provide a string that contains the date to start flagging the deprecation on (in YYYYMMDD format) and the message puppet should show. I don't currently fail the run on an expired resource but this is an option.

The are some other aspects of this to consider - Richard Clamp raised the idea of having a native type that could indicate this for an entire class (I'd rather use a function, but only because they are much easier to write) and Trevor Vaughan suggested a Puppet face that could present a report of the expired, and soon to be expired, code.

I don't know how widely useful this is but it made a nice change to write some puppet code. The small size of the example will hopefully show how easy it is to extend nearly every part of puppet - including more 'complicated' aspects like metaparameters. Although not the relationship ones, those are horrible ;) I've submitted the idea to the upstream development list so we'll see what happens.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!]]> http://infra-talk.org/2013/04/27/deprecation-warnings-from-puppet-resources/feed/ 0 http://piyush.me/2013/04/22/how-to-calculate-customer-lifetime-value-infographics/?utm_source=rss&utm_medium=rss&utm_campaign=how-to-calculate-customer-lifetime-value-infographics http://piyush.me/2013/04/22/how-to-calculate-customer-lifetime-value-infographics/?utm_source=rss&utm_medium=rss&utm_campaign=how-to-calculate-customer-lifetime-value-infographics#comments Mon, 22 Apr 2013 09:43:50 +0000 admin http://piyush.me/?p=274 http://piyush.me/2013/04/22/how-to-calculate-customer-lifetime-value-infographics/feed/ 0