Update: see this post for the latest update on getting up and running with Test-Kitchen on windows.

About a week ago I attended the 2014 Chef Summit. I got to meet a bunch of new and interesting people and also met several who I had interacted with online but had never seen in person. One new person I met was Jay Mundrawala (@jdmundrawala). Jay works for chef and built a Test-Kitchen Busser for Pester (as a personal oss contribution and not as part of his job at Chef). You might ask…a What for What? Well this post is going to attempt to answer that and explain why I think it is important.

Pester

Pester is a unit testing framework for Powershell. It was originally created by Scott Muc (@scottmuc) a few years back. I joined in in 2012 to add support for Mocking and now development has largely been taken over by Dave Wyatt (@MSH_Dave). It is a BDD style approach to writing and running unit tests for powershell. However, as we will see here, you can write more than just unit tests. You can write a suite of tests to ensure your infrastructure is built and runs as intended.

The whole idea of writing tests for powershell is new to a lot of long time scripters. However, as just mentioned, this framework has been around for a few years but is just now starting to gain some popularity among the powershell community and in fact the Powershell team at Microsoft is now beginning to use it themselves.

Many entrenched in the Chef ecosystem have undoubtedly been exposed to rspec and rspec derivative tools for writing tests for their chef recipes and other ruby gems. Pester is very much inspired by rspec and many familiar with rspec who take a first look at Pester may not immediately notice the difference. There are indeed several differences but the primary difference is one is written in and for ruby and the other powershell.

Test-Kitchen

Test kitchen is a tool that is widely used within the Chef community but can also be used by other Configuration management tools like Puppet. Test kitchen is not a test framework per se but it is a sort of meta framework that provides a plugin architecture around configuration management scripts that makes it easy to use one or more of many testing frameworks with your infrastructure management scripts.

There are issues specific to configuration management that make such a tool as Test-kitchen very useful. In addition to simply running tests, Test-Kitchen can manage the creation and destruction of a VM or other computing resource where tests can be run in a repeatable, disposable and rebuildable manner. Again, this is managed by another plugin family of provisioners. Some may use the vagrant driver, others docker, vsphere, EC2, etc. Using Test-kitchen, I can watch as an instance is provisioned, built, tested ad then destroyed without any side effects impacting my local environment.

The plugin that manages different test frameworks is called the busser. This plugin is responsible for “bussing” code from your local machine to a virtual test instance. Jay’s busser, like all the others simply make sure that Pester gets installed on the system where you want your tests to run. Since Pester is a powershell based tool. You are typically going to be running Pester tests on a windows machine and the cool thing here is that you can write them in “pure” powershell. No need to wrap all of your powershell inside of ruby language constructs. Its all 100% powershell here.

Enter DSC – Microsoft’s Desired State Configuration

This is an interesting one because it is both a product (or API) of a specific technology vendor and a long time philosophical approach to infrastructure management. Some also incorrectly interpret it as a competitor trying to unseat tools like chef or Puppet. There is indeed some overlap between DSC and other configuration management tools but the easiest way to groc how DSC fits into the CM landscape is as an API for writing resources specifically for windows infrastructure. Chef, Puppet and other tools provide a broad range of features to help you oversee and codify your infrastructure. The DSC surface area is really much simpler. DSC as it stands today consists of a constantly growing set of resources that can be leveraged in your configuration management tool of choice.

What do I mean by “resource?” Resource is a ubiquitous term in the popular CM tools used to provide an abstraction or DSL over a concrete piece of infrastructure (user, group, machine, file, firewall rule, etc) The resource descries how you want this infrastructure to look and does so in code that can be reviewed, tested, linted and source controlled.

You can use straight up DSC to execute these resources which offers a bare bones approach, or you can wrap them inside of a Chef recipe that can live alongside of non-DSC resources. Now the DSC resource for your windows roles and features, sql server HA, registry keys sits inside of your larger Chef infrastructure of nodes, environments, attributes, etc.

Chef making it easy to execute DSC resources

An initial reaction to this by many would be users of DSC is, why would I use Chef? Don’t I have to learn Ruby to work with that? Well because Chef is a full featured, mature configuration management solution, you get access to all of the great reporting, and server management features of chef. If you have a mixed windows/linux shop, you can manage everything with chef. Finally, it can be a bit unwieldy using raw DSC on its own. Before you can execute DSC resources, they must be downloaded and installed. Chef makes that super easy. And as we will see with test-kitchen, now you can plug your powershell based tests right into your chef workflow.

A real world example of executing DSC resources with chef and testing with Pester

We are going to follow a typical chef workflow of writing a cookbook to build a server. In our case it will be an IIS powered web server that hosts a Nuget package feed. Nuget is a windows package management specification very similar to ruby Gems. Its also the same specification behind windows Chocolatey packages similar to apt-get/yum/rpm for linux. Our web server will provide a rest based feed similar to rubygems.org that one can use to discover nuget packages.

Welcome to the bleeding edge

Before we get started let me point out that testing cookbooks on windows has not historically been well supported but there is more interest than ever in it today. There is very active development that is driving to make this possible but it is still not available from the latest stable version of Test-Kitchen. During this year’s Chef Summit, this exact topic was discussed. The creator and maintainer of Test-Kitchen, Fletcher Nichols was present as well as several others either interested in windows support or actively working to provide first class support for windows like Salim Afiune. I was there as well and I think everyone left with a clear understanding that this work needs to come together in a future version of Test-Kitchen in the near future. I blogged on the current state of this tooling just a couple months ago. This may be seen as a continuation of that post with a specific bend towards powershell and DSC.

I will walk you through how to get your environment configured so that you can do this testing today and I will certainly update this post once the tooling is officially released.

Environment setup

I am going to assume that you do not have any of the necessary tools needed to run through the sample cookbook I am about to show. So you can pick and choose what you need to add to your system. I am also assuming you are using the ruby embedded with chefDK. If you have another ruby versioning environment, chances are you know what to do. Note: this environment does not need to be a windows box.

ChefDK

First and foremost you need chef. The easiest way to get chef along with many of the popular tools in its ecosystem like test-kitchen is to install the Chef development kit. There are downloads available for windows, mac and several linux distributions.

Vagrant

This tutorial will use Vagrant to instantiate a machine to run the cookbook and execute the tests. You can download vagrant from VagrantUp and like chef, it has downloads for all of the popular platforms.

A hypervisor

You will need something that your vagrant flavored VM can run in. Many prefer the free and feature complete VirtualBox. If you run on windows and are currently using versions 8/2012 and above, you may use Hyper-V already on your box. Note you cannot run both on the same boot instance.

Git

You will be using git to download some of the tools I am about to mention.

The WinRM Test-Kitchen fork

This will eventually and hopefully soon be merged into the authoritative test-kitchen repo. This fork has been largely developed by Salim Afiune and can be found here. There is still active development here. Currently I have my own fork of this fork working to improve performance of winrm based file transfers. My fork hopes to dramatically improve upload times of cookbooks to the test instance. The cookbook in this tutorial should just take a couple minutes to upload using my fork compared to nearly an hour and we hope to get the perf much more faster than that. Note that WinRM has no equivalent SCP functionality so implementing this is a bit crude. Here is how you can use and install my fork:

git clone -b one_session https://github.com/mwrock/test-kitchen
copy-item test-kitchen\lib `
  C:\opscode\chefdk\embedded\apps\test-kitchen `
  -recurse -force
copy-item test-kitchen\support `
  C:\opscode\chefdk\embedded\apps\test-kitchen `
  -recurse -force
cd test-kitchen
C:\opscode\chefdk\embedded\bin\gem build test-kitchen.gemspec
C:\opscode\chefdk\embedded\bin\gem install test-kitchen-1.3.0.gem

The Winrm based Kitchen-Vagrant plugin fork

Salim has also ported his enhancements to the popular Kitchen-Vagrant Test-Kitchen plugin. Since this tutorial uses vagrant, you will need this fork. Note that if you plan to use Hyper-V or a non VirtualBox hypervisor, please use my fork that includes recent changes to make vagrant and the winrm test kitchen work outside of VirtualBox. Here is how to get and install this:

git clone -b Transport https://github.com/mwrock/kitchen-vagrant
cd kitchen-vagrant
C:\opscode\chefdk\embedded\bin\gem build kitchen-vagrant.gemspec
C:\opscode\chefdk\embedded\bin\gem install kitchen-vagrant-0.16.0.gem

The dsc_nugetserver repository containing a sample cookbook and pester tests

This can simply be cloned from https://github.com/mwrock/dsc_nugetserver.

DSC in a chef recipe

Similar to the WinRM Test-Kitchen work, the DSC recipe work done by the folks at chef is still in fairly early development. There is a dsc_script resource available in the latest chef client release as of this post. There is also a community cookbook that represents a prototype of work that will be evolved into the core chef client. This cookbook contains the dsc_resource resource.

I intentionally wrote the dsc_nugetserver cookbook almost entirely from DSC resources. Lets take a look in the default recipe and observe the two flavors of the dsc resource.

dsc_script

dsc_script  "webroot" do
  code <<-EOH
    File webroot
    {
      DestinationPath="C:\\web"
      Type="Directory"
    }
  EOH
end

This is what is currently supported by the official chef-client and ships with the latest version. They really just wrap the DSC Configuration syntax supported by powershell today. The benefit that you get using it inside of a chef recipe are that you can now use the dsc_script as just another resource in your wider library of cookbooks. Chef also does some leg work for you. You do not need to worry about where the resource is installed and you do not need to compile the resource before use.

dsc_resource

dsc_resource "http firewall rule" do
  resource_name :xfirewall
  property :name, "http"
  property :ensure, "Present"
  property :state, "Enabled"
  property :direction, "Inbound"
  property :access, "Allow"
  property :protocol, "TCP"
  property :localport, "80"
end

This is really similar if not the same as dsc_scipt just with different syntax. Note the use of the property DSL. dsc_resource also does a much better job at finding the correct resource. While I believe that dsc_script only works with the official microsoft preinstalled resources, the community dsc cookbook can locate the newer experimental resources that are being distributed as part of the community resource kit waves.

Using the resource_kit recipe to download and install all of the current resource wave kit modules

I have included a recipe that will download the latest batch of resource wave dsc resources. I basically just copied this from one of chef’s own cookbook examples and replaced the download url with the latest resource wave. Once this recipe runs, literally all dsc resources are available for you to use.

Whatif Bug affecting most resources used within chef

There is a bug in both of the dsc resource flavors that will cause most resources to crash. If the dsc resource either does not support ShouldProcess of if the underlying call to powershell DSC’s Set-TargetResource results in the function throwing an error, these chef resources currently to not provide graceful failure for these scenarios. So as is, the resource will break when called. The chef team knows about this and has a fix that will be released in a future release.

In the meantime, I have forked the community dsc_resource in the dsc cookbook and commented out a single line. I can consume this fork from any cookbook by adding this to the Berksfile:

source "https://supermarket.getchef.com"

metadata

cookbook 'dsc' , git: 'https://github.com/mwrock/dsc'

Converging the recipe

The sample cookbook comes with both a .kitchen.yml file that includes a pointer to an evaluation copy of windows 8.1 for testing. I would have included a 2012 box instead but my 2012 vagrant box is Hyper-V only and I have not had time to add virtual box.

So running:

kitchen converge

Should create a windows box for testing and converge that box to run the sample recipe.

[2014-10-13T02:34:38-07:00] INFO: Getting PowerShell DSC resource 'xfirewall'
[2014-10-13T02:35:26-07:00] INFO: DSC Resource type 'xfirewall' Configuration completed successfully
[2014-10-13T02:35:29-07:00] INFO: Chef Run complete in 534.665725 seconds
[2014-10-13T02:35:29-07:00] INFO: Removing cookbooks/dsc_nugetserver/files/default/NugetServer.zip from the cache; it is no longer needed by chef-client.
[2014-10-13T02:35:29-07:00] INFO: Running report handlers
[2014-10-13T02:35:29-07:00] INFO: Report handlers complete
Finished converging <default-windows-81> (13m6.61s).
-----> Kitchen is finished. (13m11.87s)
C:\dev\dsc_nugetserver [master]>

Note that there is a chance the kitchen converge will fail shortly after creating the box and just before downloading the chef client. My suspicion is that this is because the windows 8.1 box is hard at work installing updates and the initial winrm call times out. I have always had success immediately calling kitchen converge again.

So once this completes, you should be able to open a local browser and point at your test box to see the nuget server informational home page:

Testing the recipe with Pester

Here are the tests we will run with Pester:

describe "default recipe" {

  it "should expose a nuget packages feed" {
    $packages = Invoke-RestMethod -Uri "http://localhost/nuget/Packages"
    $packages.Count | should not be 0
    $packages[0].Title.InnerText | should be 'elmah'
  }

  context "firewall" {

    $rule = Get-NetFirewallRule | ? { $_.InstanceID -eq 'http' }
    $filter = Get-NetFirewallPortFilter | ? { $_.InstanceID -eq 'http' }

    it "should filter port 80" {
      $filter.LocalPort | should be 80
    }
    it "should be enabled" {
      $rule.Enabled | should be $true
    }
    it "should allow traffic" {
      $rule.Action | should be "Allow"
    }
    it "should apply to inbound traffic" {
      $rule.Direction | should be "Inbound"
    }    
  }
}

This is 100% powershell. No ruby to see here.

This is first going to test our nuget server website. If all went as we intended, an http call to the root of localhost should reach our nuget server and it should behave like a nuget feed. So here we expect the Packages feed to return some packages and knowing what the first package should be, we test that its name is what we expect.

Because Test-Kitchen runs tests on the converged node, we need to be sure that the outside world can reach our entry point. So we go ahead and test that we opened the firewall correctly.

kitchen verify

The kitchen Pester busser now installs Pester:

C:\dev\dsc_nugetserver [master]> kitchen verify
-----> Starting Kitchen (v1.3.0)
-----> Setting up <default-windows-81>...
       Successfully installed thor-0.19.0
       Successfully installed busser-0.6.2
       2 gems installed
       Plugin pester installed (version 0.0.6)
-----> Running postinstall for pester plugin
-----> [pester] Installing PsGet
Downloading        PsGet from https://github.com/psget/psget/raw/master/PsGet/PsGet.psm1
PsGet is installed and ready to use
       USAGE:
           PS> import-module PsGet
           PS> install-module PsUrl

       For more details:
           get-help install-module
       Or visit http://psget.net
-----> [pester] Installing Pester

Then it runs our tests:

-----> Running pester test suite
-----> [pester] Running
Executing all tests in 'C:\tmp\busser\suites\pester'Describing        default recipe
[+] should expose a nuget packages feed 4.02s   Context        firewall
[+] should filter port 80 3.18s           
[+] should be enabled 16ms
[+] should allow traffic 12ms
[+] should apply to inbound traffic 13ms
Tests completed in 7.23s
       Passed: 5 Failed: 0
       Finished verifying <default-windows-81> (0m22.55s).
-----> Kitchen is finished. (0m59.74s)
C:\dev\dsc_nugetserver [master]>

Bugs regarding Execution Policy

One issue I ran into both with the dsc_resource resource and the Pester busser was a failure to bypass the ExecutionPolicy of the Powershell.exe process. This means if no one has explicitly set an execution policy on the box which they would not have if this is a newly provisioned machine and unless this is windows server 2012R2 which implements a new default ExecutionPolicy of RemoteSigned instead of Undefined, the converge will fail complaining that the execution of scripts are not allowed to run. Since the test vagrant box used here is windows 8.1, it is susceptible to this bug.

You can work around this by setting the execution policy in the recipe as is done in the sample:

powershell_script "set execution policy" do
  code <<-EOH
    Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force
    if(Test-Path "$env:SystemRoot\\SysWOW64") {
      Start-Process "$env:SystemRoot\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe" -verb runas -wait -argumentList "-noprofile -WindowStyle hidden -noninteractive -ExecutionPolicy bypass -Command `"Set-ExecutionPolicy RemoteSigned`""
    }
    EOH
end

We set the policy for both the 64 and 32 bit shells since chef-client is a 32 bit process.

I have filed an issue with the dsc cookbook here and submitted a pull request for the busser here.

Testing on windows keeps getting better

We are still not where we need to be but we are making progress. This is a big step I think to adding accesibility to the work coming out of the Microsoft DSC initiatives. Here you have all the tools you need to not only execute DSC resources but also test them. A big thanks to Jay and Salim for their work here with Test Kitchen and the Pester busser!

If you want to learn more about DSC or Chef and particularly DSC in Chef. Pay attention to Steven Murawski's blog. Steven is a Chef Community manager and has done a ton of work with DSC at his previous employer Stack Exchange, the home of StackOverflow.com.

Hurry Up and Wait!

Tales from an automation engineer

Configure and test windows infrastructure using Powershell technologies DSC and Pester running from Chef and Test-Kitchen /October 13, 2014 by Matt Wrock