Testing Chef Cookbooks. Part 2. Chefspec.

So now you have less errors and typos in your cookbooks, thanks to foodcritic. But you are still far from confident that your cookbook will not fail to run on some node. Next step for acquiring it is unit tests (aka specs in ruby world).

Ruby has already a great spec library useful for unit testing every kind of project - it's called rspec. Many specialized unit test libraries are based on it and so is chefspec - the gem to write unit tests for your cookbooks.

Chefspec makes it easy to write unit tests for Chef recipes, get feedback fast on changes in cookbooks. So first let's install it.

sudo gem install rake chefspec --no-ri --no-rdoc

This will also add create_specs command to knife, which creates specs for particular existing cookbook:

knife cookbook create_specs my_cookbook

After this you will get a separate *_spec.rb file in my_cookbook/specs/ for every recipe file. Chefspec readme has very good examples teaching how to write tests. A couple of things I personally do different is I use subject and should instead of let(:chef_run) and expect(chef_run).to, because it allows to omit subject in some cases: (Read why RSpec developers actually recommend using expect_to syntax)

#Chefspec recommendations
describe "example::default" do
  let( :chef_run ){ ChefSpec::ChefRunner.new.converge described_recipe }
  it { expect(chef_run).to do_something }
  it 'does some other thing' do
    expect(chef_run).to do_another_thing
  end
end

#My typical specs
describe "example::default" do
  subject { ChefSpec::ChefRunner.new.converge described_recipe }
  it { should do_something }
  it 'does some other thing' do
    should do_another_thing
  end
end

We can also integrate it with Jenkins by making rspec output results in JUnit xml format that Jenkins understands. We need another gem for that:

sudo gem install rake rspec_junit_formatter --no-ri --no-rdoc

Now we can run rspec with the following parameters and it will output test results into test-results.xml:

rspec my_cookbook --format RspecJunitFormatter --out test-results.xml

Rspec also supports rake, so it may be more convenient to use it to run specs on your cookbooks:

desc 'Runs specs with chefspec.'
RSpec::Core::RakeTask.new :spec, [:cookbook, :recipe, :output_file] do |t, args|
 args.with_defaults( :cookbook => '*', :recipe => '*', :output_file => nil )
 t.verbose = false
 t.fail_on_error = false
 t.rspec_opts = args.output_file.nil? ? '--format d' : "--format RspecJunitFormatter --out #{args.output_file}"
 t.ruby_opts = '-W0' #it supports ruby options too
 t.pattern = "cookbooks/#{args.cookbook}/spec/#{args.recipe}_spec.rb"
end

Testing Chef Cookbooks. Part 1. Foodcritic.

This post have been in draft for almost a year. At last I have made myself to turn back to my blog and continue writing.

A couple of years ago we have adopted automating our server configuration through chef recipes. In the beginning we didn't have many cookbooks and all the recipes were more or less simple. But as time passed new applications had to be installed and configured, including some not so simple scenarios. Turned out that "hit and miss" method wasn't too good. We came across a lot of errors, such as:

• typo: simple as that. Although knife makes a syntax check on uploading cookbooks to server, but many times it was a wrong path or something like that, which could be revealed only after we tried to provision a node.
• missed dependencies: we ran our scripts on ec2 instances, and to save time on bootstrapping the new instance, we didn't do it every time we changed something in our scripts, but only the first time. When we finally got the recipe working on this node, sometimes it turned out that the recipe will not be working on another clean node, because we forgot some dependencies, that were somehow already installed on our test node.
• interfering applications: some recipes were used for configuring several applications. Although they were very similar, they were not identical, and sometimes changing the recipe for installing one application broke installation of the other one.

At last we came to the idea, that infrastructure code as any other code should be tested. Currently we have built a pipeline using Jenkins that first runs syntax check, then coding conventions tests, then unit tests and finally integration tests on cookbooks. And only if all the tests are passing, Jenkins runs knife cookbook upload, publishing the cookbooks on chef-server.

In my following posts I will share with you how we established this testing architecture for our chef recipes. There will 3 parts in the tutorial, we will start from the easiest checks that only make sure that your ruby code can be parsed and follows some code conventions.

First of all make sure you are running ruby 1.9.2 or newer. (We use Ubuntu on our linux servers, so all the code I provide is tested in Ubuntu 12.04 LTS.)

sudo aptitude install ruby1.9.3

Now we need rake for building our project and foodcritic for testing.

sudo gem install rake foodcritic --no-ri --no-rdoc

Chef cookbook repository has a Rakefile in it. Now when we have rake installed, we can try and run rake inside the folder Rakefile is in.

rake

This should run syntax check on your recipes, templates and files inside cookbooks. Next we can try to run foodcritic tests on your cookbooks. Type

foodcritic cookbooks

and hit enter (assuming that "cookbooks" is the folder your cookbooks are in). If it founds some warnings, it will print them out, otherwise an empty string will be printed.

Of course you may not agree with some of the rules and would like to ignore them. This could be achieved by providing additional options to foodcritic command. You can also write your own rules and add them into checks. For example, there was a FC001 rule, which stated "Use strings in preference to symbols to access node attributes". I actually preferred vice versa using symbols to strings. So I created a new rule:

rule "JT001", "Use symbols in preference to strings to access node attributes" do
 tags %w{style attributes jt}
 recipe do |ast|
  attribute_access( ast, :type => :string )
 end
end

and saved it into foodcritic-rules.rb file. Then it was easy to disable the existing FC001 rule and enable mine with:

foodcritic cookbooks --include foodcritic-rules.rb --tags ~FC001

There are also some 3^rd party rules available, so you have something to start with.

Now we will join rake and foodcritic together by creating a rake task that runs foodcritic tests. Add a new rake task similar to:

desc "Runs foodcritic linter"
task :foodcritic do
  if Gem::Version.new("1.9.2") <= Gem::Version.new(RUBY_VERSION.dup)
    sh "foodcritic cookbooks --epic-fail correctness"
  else
    puts "WARN: foodcritic run is skipped as Ruby #{RUBY_VERSION} is < 1.9.2."
  end
end
task :default => 'foodcritic'

Now when you run rake it should run both syntax and code conventions tests.

Next post will be about unit tests using chefspec.