I discovered spoon.net a few months ago from some comment on twitter. They claimed to be bringing containers to windows. Not the ability to run linux containers on windows but actual windows containers. This perked my interest and I immediately started hacking. The technology was impressive. Creating and running these containers have many similarities to workflows I'm familiar with on docker. The file system and windows registry inside these containers are truly isolated and you can layer spoon images on top of each other as part of the container packaging process.
Despite these cool features, there were some gaps that limited their usability for my scenarios. The winsxs (god bless it) was not entirely isolated limiting MSI installations and many windows features available via server manager and DISM did not fully work in isolation. I totally respect the fact that this is not an easy problem to solve.
I've kept in touch with Kenji Obata, CEO and founder of Spoon who has been filling me in on the MSI support. Being able to isolate MSI installs is key and the team at spoon has been making lots of headway making this work. This week my daughter asked me to build a minecraft server and I thought this would be a great opportunity to play with spoon to see where it has come. Running a minecraft server is fairly straight forward. You need to install the java runtime, which installs as an MSI, and then invoke java.exe to run the minecraft server .jar which starts the process running the minecraft server. The minecraft server listens for TCP requests on a specified port to connect players with game activity.
I was not only able to create a container running a minecraft server, but was later able to create an empty container and bootstrap it as a chef node then let chef install java and the minecraft jar...and then a second container running a separate instance.
This post is going to walk you through
- How to package an application inside a spoon windows container.
- Create a base container connectable via SSH
- Bootstrap containers joining them to a hosted chef server
I'l be using the minecraft server as the example app because minecraft is cool. This is gonna be a whirl wind tour where we will use several popular "devops" style apps to interact with our windows containers including vagrant, chocolatey, and chef. If you would like to follow along, I am demonstrating all of this using a windows server 2012 R2 vagrant box as the container host. This box is publicly available from vagrant cloud:
vagrant init mwrock/Windows2012R2
Its the same minimal windows box described here a few weeks ago.
Getting and using spoon
Just like you need to install the docker engine to run docker containers, you need to install the spoon plugin on a host system where you intend to run spoon containers. You can download this from spoon.net. Lets install this on our vagrant vm now from a powershell console:
Invoke-WebRequest -Uri http://start-c.spoon.net/layers/setup/3.33.8.473/spoon-plugin.exe ` -OutFile spoon-plugin.exe ./spoon-plugin.exe
You will need to open a new console for the changes to your path to take effect. Next you need to sign in to spoon. So make sure to sign up for a free spoon.net account first. I'm now gonna sign in to the spoon CLI:
spoon login mwrockx
Now we are ready to interact with the spoon CLI, creating and running containers.
Containerizing a windows based Minecraft server
There are a few ways to containerize an app in spoon. All of these are well documented on the spoon documentation site. One way is to start with a "clean" image or a collection of base images that have already been created and have components required by your app. Then manually install and configure the rest of your application and finally commit the image. All containers are created from an image. Think of the "image" as a single snapshot that holds an app setup and configuration in a pristine state. One then "runs" containers from that image. Containers are the isolated environment actually running the application.
Since minecraft requires the java runtime and there is a oracle/jre spoon image, that's a great place to start the packaging. So we'll run the java spoon image:
PS C:\Users\Administrator> spoon run oracle/jre Using VM 11.6.381 from local Using jre from local Using clean:4.0 from local Running container 4ef7018d with visibility public (use `--private` for a private container)
Spoon just launched a new command window running inside of a new container that has java installed. See:
(4ef7018d) C:\Users\Administrator>echo %java_home% C:\Program Files (x86)\java\jre7 (4ef7018d) C:\Users\Administrator>
Our java_home environment variable is set here. We won't see this variable in our host vm outside the container (assuming we have not installed java). Also note the hash visible at every prompt inside a container. This helps us to identify which container we are inside of. We can list all containers on the host with:
PS C:\Users\Administrator> spoon containers ID Images Command Created Status Visibility -- ------ ------- ------- ------ ---------- 4ef7018d spoonbrew/clean:4.0,oracle/jre 11/30/2014 9:38:42 PM Running Public c431ad42 spoonbrew/clean:4.0,freeSSHd:1.2.6 11/30/2014 12:05:16 PM Running Public 605f8124 spoonbrew/clean:4.0,freeSSHd:1.2.6 11/30/2014 3:11:20 AM Running Public
You'll see I am also running a couple of containers from a freeSSHd image. We'll get to those later. Now lets get the minecraft server .jar file. There is a different .jar for each minecraft version. I run 1.7.2 because that is the version compatible with all the minecraft mods my daughters run.
mkdir %programdata%\spoon powershell -command wget https://s3.amazonaws.com/Minecraft.Download/versions/1.7.2/minecraft_server.1.7.2.jar -outfile $env:programdata\spoon\minecraft_server.jar
Great. We have all the bits we need. So now we just need to kick off the server:
(4ef7018d) C:\Users\Administrator>"%java_home%\bin\java" -Xmx1024M -Xms1024M -ja r %programdata%\spoon\minecraft_server.jar nogui [22:01:33] [Server thread/INFO]: Starting minecraft server version 1.7.2 [22:01:33] [Server thread/INFO]: Loading properties [22:01:33] [Server thread/WARN]: server.properties does not exist [22:01:33] [Server thread/INFO]: Generating new properties file [22:01:33] [Server thread/INFO]: Default game type: SURVIVAL [22:01:33] [Server thread/INFO]: Generating keypair [22:01:33] [Server thread/INFO]: Starting Minecraft server on *:25565 [22:01:34] [Server thread/WARN]: Failed to load operators list: java.io.FileNotF oundException: .\ops.txt (The system cannot find the file specified) [22:01:34] [Server thread/WARN]: Failed to load white-list: java.io.FileNotFound Exception: .\white-list.txt (The system cannot find the file specified) [22:01:34] [Server thread/INFO]: Preparing level "world" [22:01:34] [Server thread/INFO]: Preparing start region for level 0 [22:01:35] [Server thread/INFO]: Preparing spawn area: 10% [22:01:36] [Server thread/INFO]: Preparing spawn area: 19% [22:01:37] [Server thread/INFO]: Preparing spawn area: 29% [22:01:38] [Server thread/INFO]: Preparing spawn area: 39% [22:01:39] [Server thread/INFO]: Preparing spawn area: 49% [22:01:40] [Server thread/INFO]: Preparing spawn area: 59% [22:01:41] [Server thread/INFO]: Preparing spawn area: 67% [22:01:42] [Server thread/INFO]: Preparing spawn area: 80% [22:01:43] [Server thread/INFO]: Preparing spawn area: 92% [22:01:44] [Server thread/INFO]: Done (10.273s)! For help, type "help" or "?"
Our minecraft sever is now up and running. Lets try to find the java process running from the host vm:
PS C:\Users\Administrator> ps -Name java | select path Path ---- C:\Program Files (x86)\java\jre7\bin\java.exe
You might be wondering why java is running from that path. Afterall, this java is supposed to be isolated to our container and not hanging out in our Program Files. So we sanity check with:
PS C:\Users\Administrator> Test-Path (ps -Name java | select path) False
Hmm. Its not really there. Where is it then? Pop open the windows Task Manager and look at the details tab. Notice the "Image Path Name" of java.exe. Here we find the true path:
C:\Users\Administrator\AppData\Local\Spoon\Containers\sandboxes\4ef7018d5ffa49df8253dd9a260ca016\local\stubexe\0x2118BFC595B70EF8
We can also use the spoon CLI to inspect the processes running in our containers:
PS C:\Users\Administrator> spoon ps PID Name Container User --- ---- --------- ---- 2968 java.exe 4ef7018d vagrant 4196 cmd.exe 4ef7018d vagrant
We should also see this java process listening on port 25565:
PS C:\Users\Administrator> netstat -a -o | select-string "Listening" | select-string "2968" TCP 0.0.0.0:25565 WIN-OHS0CTBB1TK:0 LISTENING 2968 TCP [::]:25565 WIN-OHS0CTBB1TK:0 LISTENING 2968
OK everything looks good and we should be able to run a minecraft client against our new server connected on that port. However, we are gonna want to stop the container before saving it as a new minecraft server spoon image. That means that any time I want to run a container off of the new image in the future, I'm going to have to make that same call to java.exe to launch the server. Wouldn't it be nice if that call was simply wrapped into the container startup? We can do that by building our image with a spoon.me file.
Automating spoon image builds with a spoon.me file
If you are used to docker, a spoon.me is the moral equivalent of a docker file. Spoon provides a small collection of scripting primitives that can be used to automate the creation of an image. This exposes some nice functionality like the "startup file" command that allows us to run a command whenever the container starts up. This also allows us to easily repeat the creation of an image and version them in source control.
Lets blow away our container where we created our minecraft server:
PS C:\Users\Administrator> spoon rm 4ef7018d Container 4ef7018d has been removed
Now lets create a file called spoon.me with the following content:
from oracle/jre
mkdir %programdata%\spoon
cmd powershell -command wget https://s3.amazonaws.com/Minecraft.Download/versions/1.7.2/minecraft_server.1.7.2.jar -outfile $env:programdata\spoon\minecraft_server.jar
startup file ("c:\Program Files (x86)\java\jre7\bin\java", "-Xmx1024M", "-Xms1024M", "-jar", "c:\programdata\spoon\minecraft_server.jar", "nogui")This script will pull down the java image, download the minecraft server jar and finally ensure that any time the image is run, the minecraft jar will be launched. Lets build our image:
spoon build -n=minecraft-server:1.7.2 spoon.me
We now have an image that we can run with:
spoon run minecraft-server
Note how fast these containers are created from our built image. Sure is faster than downloading and installing java.
Lets take our container for a spin and run a minecraft client. First we will need to open up port 25565 on the container host:
New-NetFirewallRule -DisplayName "Minecraft" `
-Direction Inbount –LocalPort 25565 `
-Protocol TCP -Action AllowI'm gonna assume you know how to run a minecraft client and point it at a server. All you need to know is the host IP address of your container host.
Bootstrapping and building spoon containers with chef
So I was super curious to see if I could integrate a spoon container into a chef workflow. Chef is a popular configuration management platform for building and maintaining infrastructure. In my mind there are two huge use cases for chef/container interaction:
- Using chef instead of (or to supplement) a spoon.me file. The chef ecosystem provides a rich set of resources just as capable of building a container image as a vm.
- Using containers to test chef recipes that will eventually be used in a vm or bare metal. As you have probably noticed, it takes much (MUCH) less time to start a container than a fresh vm.
Working with a chef server
Sure we could just use chef-zero/chef-solo to run a recipe locally, but lets try to actually perform a remote bootstrap of a container node against a serer to get the full effect here. We could use any server. Here, I'm using a hosted chef instance that I sometimes use for personal testing. Anyone can sign up for a hosted chef account giving you access to your own publicly available chef server endpoint.
See this post for a brief run through on getting started with hosted chef. I'm not going to get into the details of setting up a chef server here but you will need:
- you user .pem
- your organization validator.pem
- a knife.rb config file
Knife windows
We are going to run knife bootstrap using the knife windows gem. If you have installed chef using the ChefDK, you should have this. Otherwise you will need to run:
gem install knife-windows
You could run this directly from the container host vm. I'm just going to run it from my local os that has network access to my vm.
A minecraft cookbook with windows support
I could not find a chef cookbook for a minecraft server that runs on windows. So I wrote one and it can be found on github where you can view or clone the full cookbook. Lets take a look at the default recipe (the only recipe):
include_recipe 'chocolatey'
chocolatey 'javaruntime'
directory "#{ENV['programdata']}/minecraft"
remote_file "#{ENV['programdata']}/minecraft/minecraft_server.jar" do
source "https://s3.amazonaws.com/Minecraft.Download/versions/#{node['minecraft']['version']}/minecraft_server.#{node['minecraft']['version']}.jar"
end
cookbook_file "minecraft.bat" do
path "#{ENV['programdata']}/minecraft/minecraft.bat"
end
batch "start minecraft" do
code <<-EOH
powershell -command "start #{ENV['programdata']}/minecraft/minecraft.bat"
EOH
action :run
endA couple things worth noting:
- It uses chocolatey to install java. The java community cookbook that most use requires the java MSI to be internally hosted for windows platforms. I didn't want to bother with that and I know there is a chocolatey package that will do this job just fine. For those unfamiliar with chocolatey, it is a package management solution for windows similar to brew, yum or apt-get on mac/linux.
- I store the call to java.exe in a cookbook file, minecraft.bat, and use the chef batch resource to invoke that file via powershell Start-Process so that it does not hang the chef client run because that batch file will not return.
Upload the cookbooks to the chef server
You will need to upload the above mentioned minecraft cookbook as well as its dependencies which include:
- chef-handler
- windows
- chocolatey
All of these dependencies are on the chef supermarket. You can use berkshelf to easily download all dependencies. Again if you use the ChefDK, you have this already, otherwise install the berkshelf gem. Using the berkshelf vendor command you can grab all dependencies and save them all to a "vendor path" you specify:
berks vendor [vendor path]
Make sure your cookbook_path in your knife.rb includes the vendor path otherwise knife upload will not find them.
Establishing SSH connectivity with your containers
Currently spoon containers do not play nice with winrm and cant isolate winrm to specific containers. Spoon is aware of this and its on their radar. However, ssh works just fine and you can remotely manage your containers as long as they have an ssh server running on the container.
I have created a spoon image that runs freeSSHd. This will be the base container we will use to bootstrap with chef. You can go ahead and start up an ssh enabled container with:
spoon run -d mwrockx/freeSSHd
The -d tells spoon to run the container in the background. Note that if you do not already have this image locally (you likely don't) then spoon will download it from the public spoon hub. Now here is another cool thing about containers. Run:
spoon images
to list the images locally stored on your container host. Take a look at the size of this freeSSHd image. Its 3.7MB. That is much better than 10+GB that most windows VMs consume.
If you will be accessing the containers outside of the container host VM, you will need to allow port 22 traffic on the windows firewall:
New-NetFirewallRule -DisplayName "SSH" `
-Direction Inbount –LocalPort 22 `
-Protocol TCP -Action AllowI built this image giving both the Administrator and the vagrant users SSH access. Once the container is running, you can edit this in the freeSSHd settings by right clicking the desktop tray icon.
I'm assuming that you have access to an ssh client. I use the one that installs with msysgit located at C:\Program Files (x86)\git\bin\ssh.exe by default. Lets ssh to our container just to verify that everything works:
Yup. Looks good and the hash in the prompt correlates with my container id.
One oddity that I noticed when using ssh is that upon succesfully connecting, the ssh prompt appears at the top of the console in the midst of whatever text was already there. I'm not sure why this happens but find that calling cls to clear the screen before beginning my ssh session helps.
Bootstrap the container
ok. Enough yaking. Lets install a minecraft server with chef on a windows container:
knife bootstrap windows ssh 192.168.1.14 -P vagrant -x vagrant -r 'recipe[minecraft]' -N mc1
This will install the chef client on our container, add the container as a chef node on the server, download the minecraft cookbook and dependencies from the chef server and run the recipes to install java, the minecraft server jar file and launch the server. I use the default vagrant credentials which uses 'vagrant' for both user name and password. This will likely take a few minutes. The longest part is the download of the chef client.
I also see the node in my hosted chef organization:
Bootstrapping a second node
So just to be fancy, lets bootstrap a second node.
Networking considerations
We will have two containers running ssh and minecraft on what THEY THINK are ports 22 and 25565, we will need to setup proper NAT rules that redirect different ports coming into the container host to these ports on the containers. Spoon provides mechanisms to accomplish this. We will have our second container route ports 2222 and 35565 to 22 and 25565 respectfully. Spoon provides lots of options for mapping ports and DNS as well as linking containers to one another. See the spoon documentation for those details.
So first step we need to take is allow ports 2222 and 35565 on the container host:
Get-NetFirewallPortFilter | ? {
$_.LocalPort -eq 22
} | Set-NetFirewallPortFilter -LocalPort @(22,2222)
Get-NetFirewallPortFilter | ? {
$_.LocalPort -eq 25565
} | Set-NetFirewallPortFilter -LocalPort @(25565,35565)
Now lets run our second container informing spoon of the desired routes:
PS C:\Users\Administrator> spoon run -d --route-add=2222:22 --route-add=35565:25565 freeSSHd Using VM 11.6.381 from local Using freeSSHd:1.2.6 from local Using clean:4.0 from local Running container 9a3b62af with visibility public (use `--private` for a private container) 9a3b62af919e4ed78549b320be40df18
Just to confirm that we can ssh to the new container on port 2222:
Microsoft Windows [Version 6.3.9600]master]> ssh vagrant@192.168.1.14 -p 2222 (c) 2013 Microsoft Corporation. All rights reserved. (9a3b62af) C:\Users\Administrator>
Bootstrapping the second container
We are now ready to join the new container to the chef server and install the minecraft server recipe:
knife bootstrap windows ssh 192.168.1.14 -P vagrant -x vagrant -r 'recipe[minecraft]' -N mc2 -p 2222
Now I can see 2 nodes on the hosted server instance:
Once a node is joined to the server, we can rerun the recipes locally using chef-client. No need to bootstrap again. Lets change one of the nodes to run a different version of minecraft. We'll change the mc2 node to run 1.7.9:
Now we'll ssh to mc2 and reconverge it with the new version:
Running a minecraft client pointed to both containers, we can see the version mismatch on the second node.
Diffing containers
Here is a feature I find really clever - the ability to see how a container differs from the host. Spoon can list all files and registry keys that have changed:
PS C:\Users\Administrator> spoon run -d freeSSHd
Using VM 11.6.381 from local
Using freeSSHd:1.2.6 from local
Using clean:4.0 from local
Running container de8d95f9 with visibility public (use `--private` for a private container)
de8d95f9bd72459f81cdf543bfce5190
PS C:\Users\Administrator> spoon diff de8d95f9
Using clean:4.0 from local
Using freeSSHd:1.2.6 from local
File system changes:
C C:\Program Files (x86)
C C:\Program Files (x86)\freeSSHd
C C:\Program Files (x86)\freeSSHd\FreeSSHdService.ini
C C:\ProgramData
C C:\ProgramData\Microsoft
C C:\ProgramData\Microsoft\Windows
C C:\ProgramData\Microsoft\Windows\Caches
C C:\ProgramData\Microsoft\Windows\Caches\{6AF0698E-D558-4F6E-9B3C-3716689AF493}.2.ver0x0000000000000006.db
C C:\ProgramData\Microsoft\Windows\Caches\{8290F723-67BF-4FDA-BEEA-B0AF3ADC7010}.2.ver0x0000000000000004.db
C C:\ProgramData\Microsoft\Windows\Caches\{DDF571F2-BE98-426D-8288-1A9A39C3FDA2}.2.ver0x0000000000000004.db
C C:\ProgramData\Microsoft\Windows\Caches\cversions.2.db
C C:\Users\Administrator\AppData\Local
C C:\Users\Administrator\AppData\Local\Microsoft
C C:\Users\Administrator\AppData\Local\Microsoft\Windows
A C:\Users\Administrator\AppData\Local\Microsoft\Windows\INetCache
A C:\Users\Administrator\AppData\Local\Microsoft\Windows\INetCache\counters.dat
C C:\Users\Administrator\AppData\Local\temp
A C:\Users\Administrator\AppData\Local\temp\wodCmdTerm.exe
Registry changes:
C @HKCU@
C @HKCU@\Software
C @HKCU@\Software\Microsoft
C @HKCU@\Software\Microsoft\Windows
C @HKCU@\Software\Microsoft\Windows\CurrentVersion
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer\SessionInfo
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Explorer\SessionInfo\2
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Content
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Content\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Cookies
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Cookies\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\History
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\History\CachePrefix
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\SavedLegacySettings
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyEnable
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\AutoDetect
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\IntranetName
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\ProxyBypass
C @HKCU@\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\UNCAsIntranet
A @HKCU@\Software\Microsoft\Windows\CurrentVersion\OnDemandInterfaceCache
C @HKLM@
C @HKLM@\SOFTWARE
C @HKLM@\SOFTWARE\Wow6432Node
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows
C @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion
A @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings
A @HKLM@\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap
C @HKLM@\System
C @HKLM@\System\CurrentControlSet
A @HKLM@\System\CurrentControlSet\Services
A @HKLM@\System\CurrentControlSet\Services\Winsock
A @HKLM@\System\CurrentControlSet\Services\Winsock\Parameters
A @HKLM@\System\CurrentControlSet\Services\WinSock2
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\AppId_Catalog
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\NameSpace_Catalog5
A @HKLM@\System\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9This can come in handy for all sorts of debugging scenarios where you need to track down changes impacted by an individual install.
Are we living with rainbows and unicorns?
I'm not quite ready to confirm that but I do feel like I just got extra sprinkles added to my ice cream. There is still much to be done to fill in some isolation gaps and provide integration and orchestration tooling.
There is incredible potential here that can be taken advantage of today. Things are going to be very interesting once Microsoft reveals more details around its own container strategy and partnership with docker. However, so far that story has been very foggy and its unclear exactly what they will deliver in the next server edition. Likely many will simply wait and see what emerges there, but I think spoon has made some great strides forward here and deserves a close look for its potential as a serious player in the windows container space.