There are a few cookbook repositories I help to maintain and services like Travis and Appveyor provide a great enhancement to the pull request workflow both for contributors and maintainers. As a contributor I get fast feedback without bothering any human on whether my PR meets a minimum bar of quality and complies with the basic code style standards of the maintainers. As a maintainer, I can more easily triage incoming contributions and simply not waste time reviewing failed commits.

Its common to use travis and appveyor for unit tests and even longer running functional tests as long as they take a reasonable amount of time to run. However, in the Chef cookbook world, you usually do not see them include Test-Kitchen tests which are the cookbook equivalent of end to end functional or integration tests. I have found myself wishing for an easy way to kick off cloud based Test-Kitchen runs triggered by pull request pushes. Often there just is no way to know for sure if a commit breaks a cookbook or if it "works" unless I manually pull down the PR and invoke the kitchen test (usually via vagrant). This takes time and I wish I could just see a green check mark or red 'X' automatically without doing anything.

This week while reviewing PRs for the Chocolatey cookbook, I set out to run the kitchen tests in appveyor and did not see any readily available way to do so until Stuart Preston steered me to the proxy driver included with test-kitchen.

In this post I'll talk about some other patterns folks have used to run kitchen tests in travis, why I went with the proxy driver, and how to go about using it in your cookbook repository.

Using Docker

One approach is to install and start docker inside the travis VM and then use kitchen-docker or kitchen-dokken to run them in the travis run. This is a very viable approach. You can fire up a docker container very quickly and most cookbooks will run fine containerized. There are a few downsides:

Installing and configuring docker while fairly straight forward may require several lines of setup script.
Some cookbooks may not run well in containers.
Not a viable approach for windows based cookbooks. Looking forward to when they are.

Leveraging AWS or other cloud alternatives

Another popular approach is to run the kitchen tests from travis but reach out to another cloud provider to host the actual test instances. This also has its drawbacks:

Unless using the aws free tier, its not free and if you are, its not fast.
You have to stash keys in your travis.yml

I have an ephemeral machine, why cant I use it?

Both of the above solutions don't sit well for me because it just seems sub optimal to have to bring up another "instance" whether it be container or cloud based when travis or appveyor just did that for me. The beauty of these services is that they bring up an isolated and full fledged test VM so why do I need to bring up another one?

Most kitchen drivers are built to go "somewhere else"

This usually makes perfect sense. I don't want to run kitchen tests and have the test instance BE my machine because the state of that machine will be changed and I want to remain isolated from those changes and easily undo them. However in travis or appveyor, I AM somewhere else.

Yet having looked, I found no kitchen driver that would converge and test locally. The closest thing was kitchen-ssh which simply communicates with a test instance over SSH and does not try to spin up a vm. Surely one can easily just SSH to localhost. However, its just SSH and does not also leverage WinRM when talking to windows.

Enter the built in proxy driver

I wanted a driver that could run commands using whatever kitchen transport was configured (SSH or WinRM) but would not try to interact with a hypervisor or cloud to start a separate instance. If I just point the transport to localhost, that should succeed in running locally. Of course a "local" transport that would run native shell commands locally and use the native filesystem for file operations would be one step better. However, pointing SSH and WinRM to localhost seems to work just fine and requires no additional work.

Using the proxy driver

The chocolatey-cookbook repository includes a "real world" example that uses appveyor. I'll also briefly walk through both appveyor and travis samples here.

The .kitchen.yml or .kitchen.{travis or appveyor}.yml file

First we'll look at the .kitchen.yml file that may be the same for either travis or appveyor. Most actual cookbook repositories will likely use .kitchen.travis.yml or .kitchen.appveyor.yml since they will want to use .kitchen.yml for locally run tests.

---
driver:
  name: proxy
  host: localhost
  reset_command: "exit 0"
  port: <%= ENV["machine_port"] %>
  username: <%= ENV["machine_user"] %>
  password: <%= ENV["machine_pass"] %>

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-14.04
  - name: windows-2012R2

verifier:
  name: inspec

suites:
  - name: default
    run_list:
      - recipe[machine_test]

The driver section uses the proxy driver and configures a host, port username and password for the transport. The host setting is required and we will use "localhost". It also uses a reset_command which can be used for running a command on the instance during the "create" phase but we don't need it to do anything on a fresh appveyor or travis instance. The reset_command is required so we just specify "exit 0" which should work cross platform to do nothing.

The transport being used is determined by the same kitchen logic, either the transport is explicitly declared in the YAML or kitchen will default to SSH unless the platform name starts with "win." This example configures the credential and port from environment variables. That will be more clear when we look at the .travis.yml and appveyor.yml files.

.travis.yml

language: ruby

env:
  global:
    - machine_user=travis
    - machine_pass=travis
    - machine_port=22
    - KITCHEN_YAML=.kitchen.travis.yml

rvm:
  - 2.1.7

sudo: required
dist: trusty

before_install:
  - sudo usermod -p "`openssl passwd -1 'travis'`" travis

script:
  - bundle exec rake
  - bundle exec kitchen verify ubuntu

branches:
  only:
  - master

Here we set the port, user name and password environment variables and we set the KITCHEN_YAML variable to our special travis targeted kitchen.yml. We also give the travis user (what travis runs under) a password. Our test "script" runs kitchen verify against the ubuntu platform.

appveyor.yml

version: "master-{build}"

os: Windows Server 2012 R2
platform:
  - x64

environment:
  machine_user: test_user
  machine_pass: Pass@word1
  machine_port: 5985
  KITCHEN_YAML: .kitchen.appveyor.yml
  SSL_CERT_FILE: c:\projects\kitchen-machine\certs.pem

  matrix:
    - ruby_version: "21"

clone_folder: c:\projects\kitchen-machine
clone_depth: 1
branches:
  only:
    - master

install:
  - ps: net user /add $env:machine_user $env:machine_pass
  - ps: net localgroup administrators $env:machine_user /add
  - ps: $env:PATH="C:\Ruby$env:ruby_version\bin;$env:PATH"
  - ps: gem install bundler --quiet --no-ri --no-rdoc
  - ps: Invoke-WebRequest -Uri http://curl.haxx.se/ca/cacert.pem -OutFile c:\projects\kitchen-machine\certs.pem

build_script:
  - bundle install || bundle install || bundle install

test_script:
  - bundle exec kitchen verify windows

This one is a bit more involved but still not too bad. Like we did in the travis.yml, we assign our port, username and password environment variables and also set the .kitchen.yml file override. Here we actually go ahead and create a new user and make it an administrator. This is because its tough to get at the appveyor user's password and there may be issues changing its password in the middle of an active session.

Another thing to note for windows is we need to do some special SSL cert setup. Ruby and openssl do not use the native windows certificate store to manage certificate authorities; so we go ahead and download them and point openssl at them with the SSL_CERT_FILE variable. Those who use the Chef-dk can be thankful that hides this detail from you, but there is no chef-dk in this environment.

Finally our test_script invokes all tests in the windows platform.

The Results

Now I can see a complete Test-Kitchen log of converged resources and verifier test results right in my travis and appveyor build output.

Here is the end of the travis log:

And here is appveyor:

Hurry Up and Wait!

Tales from an automation engineer

Run Kitchen tests in Travis and Appveyor /March 6, 2016 by Matt Wrock