Creating a Windows Server 2016 Vagrant box with Chef and Packer / by Matt Wrock

I've been using Packer for a bit over a year now to create the Windows 2012 R2 Vagrant box that I regularly use for testing various server configuration scripts. My packer template has been evolving over time but is composed of some Boxstarter package setup and a few adhoc Powershell scripts. I have blogged about this process here. This has been working great, but I'm curious how it would look differently if I used Chef instead of Boxstarter and random powershell.

Chef is a much more mature configuration management platform than Boxstarter (which I would not even label as configuration management). My belief is that breaking up what I have now into Chef resources and recipes will make the image configuration more composable and easier to read. Also as an engineer employed by Chef, I'd like to be able to walk users through how this would look using Chef.

To switch things up further, I'm conducting this experimentation on a whole new OS - Windows Server 2016 TP5. This means I dont have to worry about breaking my other templates, my windows updates will be much smaller (5 updates vs > 220) and I can use DSC resources for much of the configuring. So this post will guide you through using Chef and Packer together and dealing with the "gotchas" which I ran into. The actual template can be found on github here.

If you want to "skip to the end," I have uploaded both Hyper-V and VirtualBox providers to Atlas and you can use them with vagrant via:

 vagrant init mwrock/Windows2016
 vagrant up

Preparing for the Chef Provisioner

There are a couple things that need to happen before our Chef recipes can run.

Dealing with cookbook dependencies

I've taken most of the scripts that I run in a packer run and have broken them down into various Chef recipes encapsulated in a single cookbook I include in my packer template repository. Packer's Chef provisioners will copy this cookbook to the image being built but what about other cookbooks it depends on? This cookbook uses the windows cookbook, the wsus-client cookbook and dependencies that they have and so on, but packer does not expose any mechanism for discovering those cookbooks and downloading them.

I experimented with three different approaches to fetching these dependencies. The first two really did the same thing: installed git and then cloned those cookbooks onto the image. The first method I tried did this in a simple powershell provisioner and the second method used a Chef recipe. The down sides to this approach were:

  • I had to know upfront what the exact dependency tree was and each git repo url.
  • I also would either have to solve all the versions myself or just settle for the HEAD of master for all cookbook dependencies.

Well there is a well known tool that solves these problems: Berkshelf. So my final strategy was to run berks vendor to discover the correct dependencies and their versions and download them locally to vendor/cookbooks which we ignore from source control:

C:\dev\packer-templates [master]> cd .\cookbooks\packer-templates\
C:\dev\packer-templates\cookbooks\packer-templates [master]> berks vendor ../../vendor/cookbooks
Resolving cookbook dependencies...
Fetching 'packer-templates' from source at .
Fetching cookbook index from https://supermarket.chef.io...
Using chef_handler (1.4.0)
Using windows (1.44.1)
Using packer-templates (0.1.0) from source at .
Using wsus-client (1.2.1)
Vendoring chef_handler (1.4.0) to ../../vendor/cookbooks/chef_handler
Vendoring packer-templates (0.1.0) to ../../vendor/cookbooks/packer-templates
Vendoring windows (1.44.1) to ../../vendor/cookbooks/windows
Vendoring wsus-client (1.2.1) to ../../vendor/cookbooks/wsus-client

Now I include both my packer-templates cookbook and the vendored dependent cookbooks in the chef-solo provisioner definition:

"provisioners": [
  {
    "type": "chef-solo",
    "cookbook_paths": ["cookbooks", "vendor/cookbooks"],
    "guest_os_type": "windows",
    "run_list": [
      "wsus-client::configure",
      ...

Configuring WinRM

As we will find as we make our way to a completed vagrant .box file, there are a few key places where we will need to change some machine state outside of Chef. The first of these is configuring WinRM. Before you can use either the chef-solo provisioner or a simple powershell provisioner, WinRM must be configured correctly. The Go WinRM library cannot authenticate via NTLM and so we must enable Basic Authentication and allow unencrypted traffic. Note that my template removes these settings prior to shutting down the vm before the image is exported since my testing scenarios have NTLM authentication available.

Since we cannot do this from any provisioner, we do this in the vm build step. We add a script to the <FirstLogonCommands> section of our windows answer file. This is the file that automates the initial install of windows so we are not prompted to enter things like admin password, locale, timezone, etc:

<FirstLogonCommands>
  <SynchronousCommand wcm:action="add">
     <CommandLine>cmd.exe /c C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File a:\winrm.ps1</CommandLine>
     <Order>1</Order>
  </SynchronousCommand>
</FirstLogonCommands>

The winrm.ps1 script looks like:

netsh advfirewall firewall add rule name="WinRM-HTTP" dir=in localport=5985 protocol=TCP action=allow
winrm set winrm/config/service/auth '@{Basic="true"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'

As soon as this runs on our packer build, packer will detect trhat WinRM is accessible and will move on to provisioning.

Choosing a Chef provisioner

There are two Chef flavored provisioners that come "in the box" with packer. The chef-client provisioner is ideal if you store your cookbooks on a Chef server. Since I am storing the cookbook with the packer-templates to be copied to the image, I am using the chef-solo provisioner.

Both provisioners will install the Chef client on the windows VM and will then converge all recipes included in the runlist specified in the template:

  "provisioners": [
    {
      "type": "chef-solo",
      "cookbook_paths": ["cookbooks", "vendor/cookbooks"],
      "guest_os_type": "windows",
      "run_list": [
        "wsus-client::configure",
        "packer-templates::install_ps_modules",
        "packer-templates::vbox_guest_additions",
        "packer-templates::uninstall_powershell_ise",
        "packer-templates::delete_pagefile"
      ]
    },

Windows updates and other WinRM unfriendly tasks

The Chef provisioners invoke the Chef client via WinRM. This means that all of the restrictions of WinRM apply here. That means no windows updates, no installing .net, no installing SQL server and a few other edge case restrictions.

We can work around these restrictions by isolating these unfriendly commands and running them directly via the powershell provisioner set to run "elevated":

    {
      "type": "powershell",
      "script": "scripts/windows-updates.ps1",
      "elevated_user": "vagrant",
      "elevated_password": "vagrant"
    },

When elevated credentials are used, the powershell script is run via a scheduled task and therefore runs in the context of a local user free from the fetters of WinRM. So we start by converging a Chef runlist with just enough configuration to set things up. This includes turning off automatic updates by using the wsus-client::configure recipe so that manually running updates will not interfere with automatic updates kicked off by the vm. The initial runlist also installs the PSWindowsUpdate module which we will use in the above powershell provisioner.

Here is our install-ps-modules.rb recipe that installs the Nuget package provider so we can install the PSWindowsUpdate module and the other DSC modules we will need during our packer build:

powershell_script 'install Nuget package provider' do
  code 'Install-PackageProvider -Name NuGet -Force'
  not_if '(Get-PackageProvider -Name Nuget -ListAvailable -ErrorAction SilentlyContinue) -ne $null'
end

%w{PSWindowsUpdate xNetworking xRemoteDesktopAdmin xCertificate}.each do |ps_module|
  powershell_script "install #{ps_module} module" do
    code "Install-Module #{ps_module} -Force"
    not_if "(Get-Module #{ps_module} -list) -ne $null"
  end
end

The windows-updates.ps1 looks like:

Get-WUInstall -WindowsUpdate -AcceptAll -UpdateType Software -IgnoreReboot

Multiple Chef provisioning blocks

After windows updates, I move back to Chef to finish off the provisioning:

    {
      "type": "chef-solo",
      "remote_cookbook_paths": [
        "c:/windows/temp/packer-chef-client/cookbooks-0",
        "c:/windows/temp/packer-chef-client/cookbooks-1"
      ],
      "guest_os_type": "windows",
      "skip_install": "true",
      "run_list": [
        "packer-templates::enable_file_sharing",
        "packer-templates::remote_desktop",
        "packer-templates::clean_sxs",
        "packer-templates::add_postunattend",
        "packer-templates::add_pagefile",
        "packer-templates::set_local_account_token_filter_policy",
        "packer-templates::remove_dirs",
        "packer-templates::add_setup_complete"
      ]
    },

A couple important things to include when running the Chef provisioner more than once is to tell it not to install Chef and to reuse the cookbook directories it used on the first run.

For some reason, the Chef provisioners will download and install chef regardless of whether or not Chef is already installed. Also, on the first Chef run, packer copied the cookbooks from your local environment to the vm. When it copies these cookbooks on subsequent attempts, its incredibly slow (several minutes). I'm assuming this is due to file checksum checking logic in the go library. You can avoid this sluggish file copy by just referencing the remote cookbook paths setup by the first run with the remote_cookbook_paths array shown above.

Cleaning up

Once the image configuration is where you want it to be, you might (or not) want to remove the Chef client. I try to optimize my packer setup for minimal size and the chef-client is rather large (a few hundred MB). Now you can't remove Chef with Chef. What kind of sick world would that be? So we use the powershell provisioner again to remove Chef:

Write-Host "Uninstall Chef..."
if(Test-Path "c:\windows\temp\chef.msi") {
  Start-Process MSIEXEC.exe '/uninstall c:\windows\temp\chef.msi /quiet' -Wait
}

and then clean up the disk before its exported and compacted into its final .box file:

Write-Host "Cleaning Temp Files"
try {
  Takeown /d Y /R /f "C:\Windows\Temp\*"
  Icacls "C:\Windows\Temp\*" /GRANT:r administrators:F /T /c /q  2>&1
  Remove-Item "C:\Windows\Temp\*" -Recurse -Force -ErrorAction SilentlyContinue
} catch { }

Write-Host "Optimizing Drive"
Optimize-Volume -DriveLetter C

Write-Host "Wiping empty space on disk..."
$FilePath="c:\zero.tmp"
$Volume = Get-WmiObject win32_logicaldisk -filter "DeviceID='C:'"
$ArraySize= 64kb
$SpaceToLeave= $Volume.Size * 0.05
$FileSize= $Volume.FreeSpace - $SpacetoLeave
$ZeroArray= new-object byte[]($ArraySize)
 
$Stream= [io.File]::OpenWrite($FilePath)
try {
   $CurFileSize = 0
    while($CurFileSize -lt $FileSize) {
        $Stream.Write($ZeroArray,0, $ZeroArray.Length)
        $CurFileSize +=$ZeroArray.Length
    }
}
finally {
    if($Stream) {
        $Stream.Close()
    }
}
 
Del $FilePath

What just happenned?

All of the Chef recipes, powershell scripts and packer templates can be cloned from my packer-templates github repo, but in summary, this is what they all did:

  • Installed windows
  • Installed all windows updates
  • Turned off automatic updates
  • Installed VirtualBox guest additions (only in vbox-2016.json template)
  • Uninstalled Powershell ISE (I dont use this)
  • Removed the page file from the image (it will re create itself on vagrant up)
  • Removed all windows featured not enabled
  • Enabled file sharing firewall rules so you can map drives to the vm
  • Enabled Remote Desktop and its firewall rule
  • Cleaned up the windows SxS directory of update backup files
  • Set the LocalAccountTokenFilterPolicy so that local users can remote to the vm via NTLM
  • Removes "junk" files and folders
  • Wiped all unused space on disk (might seem weird but makes the final compressed .box file smaller)

Most of this was done with Chef resources and we were also able to make ample use of DSC. For example, here is our remote_desktop.rb recipe:

dsc_resource "Enable RDP" do
  resource :xRemoteDesktopAdmin
  property :UserAuthentication, "Secure"
  property :ensure, "Present"
end

dsc_resource "Allow RDP firewall rule" do
  resource :xfirewall
  property :name, "Remote Desktop"
  property :ensure, "Present"
  property :enabled, "True"
end

Testing provisioning recipes with Test-Kitchen

One thing I've found very important is to be able to test packer provisioning scripts outside of an actual packer run. Think of this, even if you pair down your provisioning scripts to almost nothing, a packer run will always have to run through the initial windows install. Thats gonna be several minutes. Then after the packer run, you must wait out the image export and if you are using the vagrant post-provisioner, its gonna be several more minutes while the .box file is compressed. So being able to test your provisioning scripts in an isolated environment that can be spun up relatively quickly can save quite a bit of time.

I have found that working on a packer template includes three stages:

  1. Creating a very basic box with next to no configuration
  2. Testing provisioning scripts in a premade VM
  3. A full Packer run with the provisioning scripts

There may be some permutations of this pattern. For example I might remove windows update until the very end.

Test-Kitchen comes in real handy in step #2. You can also use the box produced by step #1 in your Test-Kitchen run. Depending on if I'm building Hyper-V or VirtualBox provider I'll go about this differently. Either way, a simple call to kitchen converge can be much faster than packer build.

Using kitchen-hyperv to test scripts on Hyper-V

The .kitchen.yml file included in my packer-templates repo uses the kitchen-hyperv driver to test my Chef recipes that provision the image:

---
driver:
  name: hyperv
  parent_vhd_folder: '../../output-hyperv-iso/virtual hard disks'
  parent_vhd_name: packer-hyperv-iso.vhdx

If I'm using a hyperv builder to first create a minimal image, packer puts the build .vhdx file in output-hyperv-iso/virtual hard disks. I can use kitchen-hyperv and point it at that image and it will create a new VM using that vhdx file as the parent of a new differencing disk where I can test my recipes. I can then have test-kitchen run these recipes in just a few minutes or less which is a much tighter feedback loop than packer provides.

Using kitchen-vagrant to test on Virtualbox

If you create a .box file with a minimal packer template, it will output that .box file in the root of the packer-template repo. You can add that box to your local vagrant repo by running:

vagrant box add 2016 .\windows2016min-virtualbox.box

Now you can test against this with a test-kitchen driver config that looks like:

---
driver:
  name: vagrant
  box: 2016

Check out my talk on creating windows vagrant boxes with packer at Hashiconf!

I'll be talking on this topic next month (September 2016) at Hashiconf. You can use my discount code SPKR-MWROCK for 15% off General Admission tickets.