Testing Chef Cookbooks. Part 2.5. Speeding Up ChefSpec Run

Disclaimer: as of Jan 4 2014, this is already implemented inside ChefSpec (>= 3.1.2), so you don't have to do anything. The post just describes the problem and solution with more details.

Last time we were speaking about testing Chef recipes, I introduced to you ChefSpec as a very good tool for running unit tests on your cookbooks. But lately I encountered a problem with it. I have over 800 unit tests (aka examples in RSpec world) and now it takes about 20 minutes to run. 20 minutes!!! That's a extremely long time for this kind of task. So I decided to delve, what exactly is responsible for taking so much time.

My examples look like that (many recipes have similar example groups for windows and mac_os_x):

describe "example::default" do
  context 'ubuntu' do
    subject { ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    let( :node ) { subject.node }

    it { should do_something }
    it 'does some other thing' do
      should do_another_thing
    end
    it { should do_much_more }
  end
end

I put some printouts inside describe, context, subject and let blocks, as well as read RSpec documentation about let and subject. Turned out, that subject and let blocks are called for every test, i.e. they are cached when accessed inside 1 test (it block), but not across tests inside test group (in our case ubuntu context). So for these tests subject is actually calculated 3 times. That is not a problem for ordinary RSpec tests, where subject most of the time is an object returned by constructor, e.g. User.new. But in ChefSpec case we have a converge operation as Subject under Test (SuT), which is more costly and takes more time to calculate. Another difference is that, opposing to ordinary RSpec tests we do not change the SuT in ChefSpec, but just make sure that it has right resources with right actions. So running converge for every example is a huge overhead.

How can we fix that? Well, obviously we should somehow save the value across the examples. I tried different approaches, some of them worked partially, some didn't at all. The simplest thing was to use before :all block.

describe "example::default" do
  context 'ubuntu' do
    before :all { @chef_run = ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    subject { @chef_run }
    [...]
  end
end

It does not require any more than small change in spec files, but the drawback of this approach is no mocking is supported in before :all block. So if you have to mock for example file existence, it would not work:

describe "example::default" do
  context 'ubuntu' do
    before :all do
      ::File.stub( :exists? ).with( '/some/path/' ).and_return false
      @chef_run = ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe
    end
    subject { @chef_run }
    [...]
  end
end

RSpec allows to extend modules with your own methods and the idea was to write method similar to let, but which will cache the results across examples too. Create a spec_helper.rb file somewhere in your Chef project and add the following lines there:

module SpecHelper
  @@cache = {}
  FINALIZER = lambda {|id| @@cache.delete id }

  def shared( name, &block )
    location = ancestors.first.metadata[:example_group][:location]
    define_method( name ) do
      unless @@cache.has_key? Thread.current.object_id
        ObjectSpace.define_finalizer Thread.current, FINALIZER
      end
      @@cache[Thread.current.object_id] ||= {}
      @@cache[Thread.current.object_id][location + name.to_s] ||= instance_eval( &block )
    end
  end

  def shared!( name, &block )
    shared name, &block
    before { __send__ name }
  end
end

RSpec.configure do |config|
  config.extend SpecHelper
end

Values from @@cache are never deleted, and you can use same names with this block, so I also use location of the usage, which looks like that: "./cookbooks/my_cookbook/spec/default_spec.rb:3". Now change subject into shared( :subject ) in your specs:

describe "example::default" do
  context 'ubuntu' do
    shared( :subject ) { ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    [...]
  end
end

And when running the tests you will now have to include the spec_helper.rb too:

rspec --include ./relative/path/spec_helper.rb cookbooks/*/spec/*_spec.rb

If you use the rake task I introduced in previous post, add the following line to it.

desc 'Runs specs with chefspec.'
RSpec::Core::RakeTask.new :spec, [:cookbook, :recipe, :output_file] do |t, args|
  [...]
  t.rspec_opts += ' --require ./relative/path/spec_helper.rb'
  [...]
end

And that's all! Now tests run in 2 minutes. 10 times faster!

Extracting Files From tar.gz With Ruby

I always thought that it should be a trivial task. There are even some stackoverflow answers on that topic, but there is actually a catch that none of the answers talks about. Originally tar did not support paths longer than 100 chars. GNU tar is better and they implemented support for longer paths, but it was made through a hack called ././@LongLink. Shortly speaking, if you stumble upon an entry in tar archive which path equals to above mentioned ././@LongLink, that means that the following entry path is longer than 100 chars and is truncated. The full path of the following entry is actually the value of the current entry. So when extracting files from tar we also must have in mind this possibility.

require 'rubygems/package'
require 'zlib'

TAR_LONGLINK = '././@LongLink'
tar_gz_archive = '/path/to/archive.tar.gz'
destination = '/where/extract/to'

Gem::Package::TarReader.new( Zlib::GzipReader.open tar_gz_archive ) do |tar|
  dest = nil
  tar.each do |entry|
    if entry.full_name == TAR_LONGLINK
      dest = File.join destination, entry.read.strip
      next
    end
    dest ||= File.join destination, entry.full_name
    if entry.directory?
      FileUtils.rm_rf dest unless File.directory? dest
      FileUtils.mkdir_p dest, :mode => entry.header.mode, :verbose => false
    elsif entry.file?
      FileUtils.rm_rf dest unless File.file? dest
      File.open dest, "wb" do |f|
        f.print entry.read
      end
      FileUtils.chmod entry.header.mode, dest, :verbose => false
    elsif entry.header.typeflag == '2' #Symlink!
      File.symlink entry.header.linkname, dest
    end
    dest = nil
  end
end

Testing Chef Cookbooks. Part 2. Chefspec.

So now you have less errors and typos in your cookbooks, thanks to foodcritic. But you are still far from confident that your cookbook will not fail to run on some node. Next step for acquiring it is unit tests (aka specs in ruby world).

Ruby has already a great spec library useful for unit testing every kind of project - it's called rspec. Many specialized unit test libraries are based on it and so is chefspec - the gem to write unit tests for your cookbooks.

Chefspec makes it easy to write unit tests for Chef recipes, get feedback fast on changes in cookbooks. So first let's install it.

sudo gem install rake chefspec --no-ri --no-rdoc

This will also add create_specs command to knife, which creates specs for particular existing cookbook:

knife cookbook create_specs my_cookbook

After this you will get a separate *_spec.rb file in my_cookbook/specs/ for every recipe file. Chefspec readme has very good examples teaching how to write tests. A couple of things I personally do different is I use subject and should instead of let(:chef_run) and expect(chef_run).to, because it allows to omit subject in some cases: (Read why RSpec developers actually recommend using expect_to syntax)

#Chefspec recommendations
describe "example::default" do
  let( :chef_run ){ ChefSpec::ChefRunner.new.converge described_recipe }
  it { expect(chef_run).to do_something }
  it 'does some other thing' do
    expect(chef_run).to do_another_thing
  end
end

#My typical specs
describe "example::default" do
  subject { ChefSpec::ChefRunner.new.converge described_recipe }
  it { should do_something }
  it 'does some other thing' do
    should do_another_thing
  end
end

We can also integrate it with Jenkins by making rspec output results in JUnit xml format that Jenkins understands. We need another gem for that:

sudo gem install rake rspec_junit_formatter --no-ri --no-rdoc

Now we can run rspec with the following parameters and it will output test results into test-results.xml:

rspec my_cookbook --format RspecJunitFormatter --out test-results.xml

Rspec also supports rake, so it may be more convenient to use it to run specs on your cookbooks:

desc 'Runs specs with chefspec.'
RSpec::Core::RakeTask.new :spec, [:cookbook, :recipe, :output_file] do |t, args|
 args.with_defaults( :cookbook => '*', :recipe => '*', :output_file => nil )
 t.verbose = false
 t.fail_on_error = false
 t.rspec_opts = args.output_file.nil? ? '--format d' : "--format RspecJunitFormatter --out #{args.output_file}"
 t.ruby_opts = '-W0' #it supports ruby options too
 t.pattern = "cookbooks/#{args.cookbook}/spec/#{args.recipe}_spec.rb"
end

Testing Chef Cookbooks. Part 1. Foodcritic.

This post have been in draft for almost a year. At last I have made myself to turn back to my blog and continue writing.

A couple of years ago we have adopted automating our server configuration through chef recipes. In the beginning we didn't have many cookbooks and all the recipes were more or less simple. But as time passed new applications had to be installed and configured, including some not so simple scenarios. Turned out that "hit and miss" method wasn't too good. We came across a lot of errors, such as:

• typo: simple as that. Although knife makes a syntax check on uploading cookbooks to server, but many times it was a wrong path or something like that, which could be revealed only after we tried to provision a node.
• missed dependencies: we ran our scripts on ec2 instances, and to save time on bootstrapping the new instance, we didn't do it every time we changed something in our scripts, but only the first time. When we finally got the recipe working on this node, sometimes it turned out that the recipe will not be working on another clean node, because we forgot some dependencies, that were somehow already installed on our test node.
• interfering applications: some recipes were used for configuring several applications. Although they were very similar, they were not identical, and sometimes changing the recipe for installing one application broke installation of the other one.

At last we came to the idea, that infrastructure code as any other code should be tested. Currently we have built a pipeline using Jenkins that first runs syntax check, then coding conventions tests, then unit tests and finally integration tests on cookbooks. And only if all the tests are passing, Jenkins runs knife cookbook upload, publishing the cookbooks on chef-server.

In my following posts I will share with you how we established this testing architecture for our chef recipes. There will 3 parts in the tutorial, we will start from the easiest checks that only make sure that your ruby code can be parsed and follows some code conventions.

First of all make sure you are running ruby 1.9.2 or newer. (We use Ubuntu on our linux servers, so all the code I provide is tested in Ubuntu 12.04 LTS.)

sudo aptitude install ruby1.9.3

Now we need rake for building our project and foodcritic for testing.

sudo gem install rake foodcritic --no-ri --no-rdoc

Chef cookbook repository has a Rakefile in it. Now when we have rake installed, we can try and run rake inside the folder Rakefile is in.

rake

This should run syntax check on your recipes, templates and files inside cookbooks. Next we can try to run foodcritic tests on your cookbooks. Type

foodcritic cookbooks

and hit enter (assuming that "cookbooks" is the folder your cookbooks are in). If it founds some warnings, it will print them out, otherwise an empty string will be printed.

Of course you may not agree with some of the rules and would like to ignore them. This could be achieved by providing additional options to foodcritic command. You can also write your own rules and add them into checks. For example, there was a FC001 rule, which stated "Use strings in preference to symbols to access node attributes". I actually preferred vice versa using symbols to strings. So I created a new rule:

rule "JT001", "Use symbols in preference to strings to access node attributes" do
 tags %w{style attributes jt}
 recipe do |ast|
  attribute_access( ast, :type => :string )
 end
end

and saved it into foodcritic-rules.rb file. Then it was easy to disable the existing FC001 rule and enable mine with:

foodcritic cookbooks --include foodcritic-rules.rb --tags ~FC001

There are also some 3^rd party rules available, so you have something to start with.

Now we will join rake and foodcritic together by creating a rake task that runs foodcritic tests. Add a new rake task similar to:

desc "Runs foodcritic linter"
task :foodcritic do
  if Gem::Version.new("1.9.2") <= Gem::Version.new(RUBY_VERSION.dup)
    sh "foodcritic cookbooks --epic-fail correctness"
  else
    puts "WARN: foodcritic run is skipped as Ruby #{RUBY_VERSION} is < 1.9.2."
  end
end
task :default => 'foodcritic'

Now when you run rake it should run both syntax and code conventions tests.

Next post will be about unit tests using chefspec.

Ruby Pack/Unpack

Having a party in ZeroTurnaround new office in Tartu. There is a mat on the floor near the entrance door that says:

01010111011001010110110001100011011011110110110101100101

Using ruby, we can quickly figure out, what that actually means:

["01010111011001010110110001100011011011110110110101100101"].pack('B*') #==> Welcome

Ruby string pack unpack detailed usage.

Google Code Jam 2011 Qualification Round

This Saturday Google Code Jam 2011 Qualification Round took place. I didn't have much time to spend on the problems, but still have solved a couple and proceeded to Round 1.

The ones that I solved are Magicka and Candy Splitting.

Magicka

First I tried to solve it with some cunning string substitution involving regular expressions. But in the end the simple simulation did the trick. I actually was a little bit disappointed after the official solutions were announced, because I thought that there should be some trick in here.

#!/usr/bin/ruby

lines = ARGF.readlines

t = lines.first.to_i
(1..t).each do |test_id|
  arr = lines[test_id].split " "
  c = arr[0].to_i
  combines = arr[1..c].inject({}){|memo, e| memo[e[0..1]]=e[2]; memo[e[1]+e[0]]=e[2]; memo }
  d = arr[c+1].to_i
  opposed = arr[c+2..c+1+d]
  n = arr[c+2+d].to_i
  elems = arr[c+3+d] #.split("").collect {|i| i.to_sym }

  result = []
  elems.split("").each do |elem|
    result << elem
    if result.length > 1
      last2 = result[-2..-1].join
      if combines.key?(last2)
        result.pop(2)
        result = result << combines[last2]
      end
      opposed.each do |o|
        if result.include?(o[0]) && result.include?(o[1])
          result = []
          break
        end
      end
    end
  end

  puts "Case ##{test_id}: [#{result.join(", ")}]"
end

Candy Splitting

This one was a little bit more tricky and the author's solution is much more simpler and clear than mine. I made it using old school brute force, just like Goro (strength is my strength :) )

#!/usr/bin/ruby

def s( candies, pile1, pile2 )
 #p "c:#{candies}, 1:#{pile1}, 2:#{pile2}"
 if candies.empty?
  if !pile1.empty? && !pile2.empty? && pile1.inject(0){|memo, i| memo ^ i} == pile2.inject(0){|memo, i| memo ^ i}
   return pile1
  else
   return false
  end
 end
 x = candies.shift
 p1 = pile1.clone
 return s( candies, p1 << x, pile2) || s( candies, pile1, pile2 << x)
end

lines = ARGF.readlines

t = lines.first.to_i
(1..t).each do |test_id|
 n_candies = lines[test_id*2-1]
 candies = lines[test_id*2].split(" ").collect{|i| i.to_i }

 r = s( candies.sort.reverse, [], [] )
 r = !r ? "NO" : r.inject(:+)

 puts "Case ##{test_id}: #{r}"
end

Google Code Jam 2010 - Round 1

Sadly I didn't qualify into Round 2. But actually didn't have any chance, because at the time I posted my first solution to the A problem, the top scorers board was already full of 100s (That is maximum one could get for solving all 3 problems.) So as the qualifying round this one had 3 problems: A, B and C, with A being the simplest one and C the hardest one. As I participated in Round 1B, here are the thoughts on problems that I met there.

File Fix-It

(Problem Text)

This one is about creating the tree of folders and counting how many folders you have to create. The solution algorithm is simple.

1. We create a root node.
2. Read the next folder, we must create (with path).
3. Split the path.
4. Add folders one by one to the root node, counting how many of them we had to create.
5. Goto 2

Here is my c++ code.

#include <iostream>
#include <vector>
#include <map>

using namespace std;

class Node{
public:
  string name;
  map<string, Node> children;

  Node(){}
  Node( string n ):name(n){}
};

vector<string> split( const string& s, const string& delimiter ){
  vector<string> result;
  string::size_type from = 0;
  string::size_type to = 0;

  while ( to != string::npos ){
    to = s.find( delimiter, from );
    if ( from < s.size() && from != to ){
      result.push_back( s.substr( from, to - from ) );
    }
    from = to + delimiter.size();
  }
  return result;
}

int add_to_node( Node* current, vector<string>& path ){
  int r = 0;
  for (int i=0; i<path.size(); i++){
    if (current->children.find(path[i]) == current->children.end()){
      r++;
      Node child(path[i]);
      current->children[path[i]] = child;
    }
    current = &( current->children[path[i]] );
  }
  return r;
}

int main(){
  int t;
  cin >> t;
  for (int i=1; i<=t; i++){
    int n, m;
    cin >> n >> m;
    Node root("");
    for (int j=0; j<n; j++){
      string s;
      cin >> s;
      vector<string> path = split( s, "/" );
      //cout << root.children.size();
      add_to_node( &root, path );
    }
    int total = 0;
    for (int j=0; j<m; j++){
      string s;
      cin >> s;
      vector<string> path = split( s, "/" );
      total += add_to_node( &root, path );
    }
    cout << "Case #" << i << ": " << total <<endl;
  }
}

Picking Up Chicks

(Problem Text)

This problem is about finding out how many (slow) chicks prevent one particular (fast) chick to get to the barn in time. (This solution is in ruby). My solution actually is not optimal, but it is still solving the problem.

#!/usr/bin/env ruby

c = ARGF.readline.to_i
(1..c).each do |i|
  n, k, b, t = ARGF.readline.split(" ").collect{|x| x.to_i }
  x = ARGF.readline.split(" ").collect{|y| y.to_i }
  v = ARGF.readline.split(" ").collect{|y| y.to_i }

  times = []
  x.each_index do |j|
    d = b-x[j]
    times << ( d % v[j] == 0 ? d / v[j] : d / v[j] + 1 )
  end

  chicks = 0
  max = 0
  swaps = 0

  times.reverse!
  times.each_index do |ti|
    max = times[ti] > max ? times[ti] : max
    if t >= max
      chicks += 1
    elsif t >= times[ti]
      chicks += 1
      #we need to swap only with those chicks that will arrive later and won't get in time
      swaps += times[0...ti].collect{|one| one > times[ti] && one > t ? one : nil }.compact.count
    end
    break if chicks >= k
  end

  print( "Case #", i,": ", (chicks >= k ? swaps : "IMPOSSIBLE"), "\n" )
end

Your Rank is Pure

(Problem Text)

I ran out of time before I even could understand what this problem was all about. So i cannot tell anything about it. I will try to solve it later though.

I have also checked out the official solutions for the first 2 problems, and they appear to be much cleaner and simpler than mine. My approach is too straightforward. Have to practice more...

Try Ruby Online

Interactive ruby console is available at http://tryruby.org/. It is not very convenient to delete wrong entered character there (you should put a cursor on it, not after it, and press backspace), but it works! :)

Ruby Shoes

I wanted to try ruby Shoes already long ago, but just didn't know what to start with. Until I found a quiteuseful article about making a game on ruby shoes. I tried it and it was unexpectedly easy. Here are some my subjective thoughts about it.

Good things:

It is very easy to study.

One can begin writing using shoes quite quickly and straightforward. No big manuals or hardly understandable api and howtos.

It is cute. :)

Yes, I really liked it. Different objects are created very easy, such as:

Shoes.app { button("Click me!") { alert("Good job.") } }

You also can draw with provided rect, oval or even arrow methods or import an image as a background. Motion ability is also provided:

Shoes.app do #A star that moves after the mouse pointer.
  @shape = star :points => 5
  motion do |left, top|
    @shape.move left, top
  end
end

Unfortunately there were also some disadvantages:

The whole Shoes application should be in Shoes.app block.

Seem to me to be a little bit uncomfortable.

You have to pass everywhere app variable that denotes the Shoes application.

Well, not just everywhere... But if you create your own class, which objects have to be drawn you also have to pass the application variable, or make it global, which is not really good pattern.

class RedRect

  def initialize( app )
    app.fill red
    app.rect :left => 10, :top => 10, :width => 40 
  end
end

Shoes.app do
  RedRect.new( self )
end

Pity, but it is slow.

The resulting snake in the game responds in about 2 seconds. It is a long period of time, especially on higher levels with higher speed.

In conclusion the authors' words about Shoes are right:"Shoes is a tiny graphics toolkit, designed for beginners". It can be used to create quickly a small application, where speed is not essential.

Hobo for Rails 2

Пока я собирался попробовать это Хобо на вкус, они уже выпустили следущую версию. А пока времени на это нет вообще.

Hobo for Rails

На днях был обнаружен плагин для Rails, который значительно расширяет возможности. Скоро напишу конкретнее, а пока только сайт проекта.

Rails database table names: singular Vs. plural

Rails < 2.0 : You need to add into config/environment.rb file line

# Include your application configuration below
ActiveRecord::Base.pluralize_table_names = false #added line

Rails >= 2.0 : You need to add into config/environment.rb file inside block

Rails::Initializer.run do |config|
  config.active_record.pluralize_table_names = false #added line
end

Install Ruby Gem Postgres In Ubuntu Linux

Installing ruby gem postgres in Ubuntu linux

sudo gem install postgres

gives the next error

ERROR:  While executing gem ... (Gem::Installer::ExtensionBuildError)
  ERROR: Failed to build gem native extension.

ruby extconf.rb ins postgres
extconf.rb:46: command not found: pg_config --includedir
extconf.rb:53: command not found: pg_config --libdir
checking for main() in -lpq... no

Can be solved by:

sudo apt-get install libpq-dev
sudo gem install postgres-pr