Testing Chef Cookbooks. Part 2.5. Speeding Up ChefSpec Run

Disclaimer: as of Jan 4 2014, this is already implemented inside ChefSpec (>= 3.1.2), so you don't have to do anything. The post just describes the problem and solution with more details.

Last time we were speaking about testing Chef recipes, I introduced to you ChefSpec as a very good tool for running unit tests on your cookbooks. But lately I encountered a problem with it. I have over 800 unit tests (aka examples in RSpec world) and now it takes about 20 minutes to run. 20 minutes!!! That's a extremely long time for this kind of task. So I decided to delve, what exactly is responsible for taking so much time.

My examples look like that (many recipes have similar example groups for windows and mac_os_x):

describe "example::default" do
  context 'ubuntu' do
    subject { ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    let( :node ) { subject.node }

    it { should do_something }
    it 'does some other thing' do
      should do_another_thing
    end
    it { should do_much_more }
  end
end

I put some printouts inside describe, context, subject and let blocks, as well as read RSpec documentation about let and subject. Turned out, that subject and let blocks are called for every test, i.e. they are cached when accessed inside 1 test (it block), but not across tests inside test group (in our case ubuntu context). So for these tests subject is actually calculated 3 times. That is not a problem for ordinary RSpec tests, where subject most of the time is an object returned by constructor, e.g. User.new. But in ChefSpec case we have a converge operation as Subject under Test (SuT), which is more costly and takes more time to calculate. Another difference is that, opposing to ordinary RSpec tests we do not change the SuT in ChefSpec, but just make sure that it has right resources with right actions. So running converge for every example is a huge overhead.

How can we fix that? Well, obviously we should somehow save the value across the examples. I tried different approaches, some of them worked partially, some didn't at all. The simplest thing was to use before :all block.

describe "example::default" do
  context 'ubuntu' do
    before :all { @chef_run = ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    subject { @chef_run }
    [...]
  end
end

It does not require any more than small change in spec files, but the drawback of this approach is no mocking is supported in before :all block. So if you have to mock for example file existence, it would not work:

describe "example::default" do
  context 'ubuntu' do
    before :all do
      ::File.stub( :exists? ).with( '/some/path/' ).and_return false
      @chef_run = ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe
    end
    subject { @chef_run }
    [...]
  end
end

RSpec allows to extend modules with your own methods and the idea was to write method similar to let, but which will cache the results across examples too. Create a spec_helper.rb file somewhere in your Chef project and add the following lines there:

module SpecHelper
  @@cache = {}
  FINALIZER = lambda {|id| @@cache.delete id }

  def shared( name, &block )
    location = ancestors.first.metadata[:example_group][:location]
    define_method( name ) do
      unless @@cache.has_key? Thread.current.object_id
        ObjectSpace.define_finalizer Thread.current, FINALIZER
      end
      @@cache[Thread.current.object_id] ||= {}
      @@cache[Thread.current.object_id][location + name.to_s] ||= instance_eval( &block )
    end
  end

  def shared!( name, &block )
    shared name, &block
    before { __send__ name }
  end
end

RSpec.configure do |config|
  config.extend SpecHelper
end

Values from @@cache are never deleted, and you can use same names with this block, so I also use location of the usage, which looks like that: "./cookbooks/my_cookbook/spec/default_spec.rb:3". Now change subject into shared( :subject ) in your specs:

describe "example::default" do
  context 'ubuntu' do
    shared( :subject ) { ChefSpec::ChefRunner.new( :platform => 'ubuntu', :version => '12.04' ).converge described_recipe }
    [...]
  end
end

And when running the tests you will now have to include the spec_helper.rb too:

rspec --include ./relative/path/spec_helper.rb cookbooks/*/spec/*_spec.rb

If you use the rake task I introduced in previous post, add the following line to it.

desc 'Runs specs with chefspec.'
RSpec::Core::RakeTask.new :spec, [:cookbook, :recipe, :output_file] do |t, args|
  [...]
  t.rspec_opts += ' --require ./relative/path/spec_helper.rb'
  [...]
end

And that's all! Now tests run in 2 minutes. 10 times faster!

Testing Chef Cookbooks. Part 2. Chefspec.

So now you have less errors and typos in your cookbooks, thanks to foodcritic. But you are still far from confident that your cookbook will not fail to run on some node. Next step for acquiring it is unit tests (aka specs in ruby world).

Ruby has already a great spec library useful for unit testing every kind of project - it's called rspec. Many specialized unit test libraries are based on it and so is chefspec - the gem to write unit tests for your cookbooks.

Chefspec makes it easy to write unit tests for Chef recipes, get feedback fast on changes in cookbooks. So first let's install it.

sudo gem install rake chefspec --no-ri --no-rdoc

This will also add create_specs command to knife, which creates specs for particular existing cookbook:

knife cookbook create_specs my_cookbook

After this you will get a separate *_spec.rb file in my_cookbook/specs/ for every recipe file. Chefspec readme has very good examples teaching how to write tests. A couple of things I personally do different is I use subject and should instead of let(:chef_run) and expect(chef_run).to, because it allows to omit subject in some cases: (Read why RSpec developers actually recommend using expect_to syntax)

#Chefspec recommendations
describe "example::default" do
  let( :chef_run ){ ChefSpec::ChefRunner.new.converge described_recipe }
  it { expect(chef_run).to do_something }
  it 'does some other thing' do
    expect(chef_run).to do_another_thing
  end
end

#My typical specs
describe "example::default" do
  subject { ChefSpec::ChefRunner.new.converge described_recipe }
  it { should do_something }
  it 'does some other thing' do
    should do_another_thing
  end
end

We can also integrate it with Jenkins by making rspec output results in JUnit xml format that Jenkins understands. We need another gem for that:

sudo gem install rake rspec_junit_formatter --no-ri --no-rdoc

Now we can run rspec with the following parameters and it will output test results into test-results.xml:

rspec my_cookbook --format RspecJunitFormatter --out test-results.xml

Rspec also supports rake, so it may be more convenient to use it to run specs on your cookbooks:

desc 'Runs specs with chefspec.'
RSpec::Core::RakeTask.new :spec, [:cookbook, :recipe, :output_file] do |t, args|
 args.with_defaults( :cookbook => '*', :recipe => '*', :output_file => nil )
 t.verbose = false
 t.fail_on_error = false
 t.rspec_opts = args.output_file.nil? ? '--format d' : "--format RspecJunitFormatter --out #{args.output_file}"
 t.ruby_opts = '-W0' #it supports ruby options too
 t.pattern = "cookbooks/#{args.cookbook}/spec/#{args.recipe}_spec.rb"
end