Dan Wanek's notes when first developing the ruby implementation of NTLM.
This week the WinRM ruby gem version 1.6.0 was released and there is really just one new feature it delivers but it is significant: NTLM/Negotiate authentication. Simply stated this provides a safe, low friction authentication and encryption mechanism long available in native Windows remote management tooling but absent in cross platform tools that implement the WinRM protocol.
In this post I'll highlight why I think this is a big deal, share a little history behind the feature release and discuss why authentication/encryption over WinRM has historically been a problem in the cross platform ecosystem.
Tell me again? Why is this awesome?
This is awesome because it means that assuming you are connecting to a Windows Server OS of Windows 2012 R2 or later, you can now securely connect from either a Windows or Linux application leveraging the ruby WinRM gem without any preconfiguration of the target machine. Client OS SKUs (like Windows 7, 8 and 10) and server versions 2008 R2 and prior still need to have WinRM explicitly enabled.
No SSL setup and no WinRM configuration that compromises its security. Just to be clear SSL is still a good idea and encouraged for production nodes, but if you are just trying to get your feet wet with Windows remote execution using tools like Chef or Vagrant, it is now much easier to accomplish and keep the security bar at a sane level.
How did this come to be?
I am a developer at Chef and we use the WinRM gem inside of Knife-Windows in order to execute remote commands on a Windows node from either Windows or Linux. We use a monkey patch gem called winrm-s to provide Negotiate authentication and encryption BUT it only works from windows workstations because it leverages the native Win32 APIs available only on Windows. Also its just monkey patches on top of the winrm and httpclient gems and therefore quite fragile.
My initial objective was to simply port the patches in winrm-s to winnrm and possibly the httpclient gem so that we could provide the same functionality but the implementation would live downstream where it belongs.
Well I remembered I had seen a PR in the WinRM repo that mentioned providing Negotiate auth implementation. It was submited by Dan Wanek, the original author of the WinRM gem. The PR was from late 2014 and I never really looked at it but filed it away in my mind as something worth looking at when the time came to drop winrm-s. So the time came and I quickly noticed that it did not seem to use any native Win32 APIs but leveraged another gem, rubyntlm (also authored by Dan Wanek), which appeared to be a pure ruby implementation of NTLM/Negotiate. That means it would work on Linux in addition to Windows which would be really really great.
It was pretty straight forward to get working and needed just a small amount of tweaking to be production ready - impressive since it had been dormant for over a year. Once it was up and running I verified that it indeed worked both from Windows and Linux. Nice work Dan!
The cross platform Windows remote execution landscape
I've written several posts covering different aspects of WinRM. Like this one and this one and also this one. Its a protocol that many native windows users likely take for granted and perhaps even forget they are using it when they are using its more full featured cousin: Powershell Remoting. However those who dont have access to direct Powershell because they either run on other operating systems that need to talk to Windows machines or are on Windows but use tools that are portable to non windows platforms are likely using a library that implements the WinRM protocol.
There are many such libraries:
- The ruby WinRM gem (covered in this post)
- WinRM for GO
- pywinrm python library
- Xebialabs Overthere Java library for remote execution
This is just a list of the most popular libraries but there are many more. These are used by well known "DevOps" automation tools such as Chef, Ansible, Packer, Vagrant and others.
WinRM is a simple SOAP based client/server protocol. So the above libraries merely implement a web service client that issues requests to a WinRM Windows service and interprets the responses. Basically these exchanges result in:
- Creating a Shell
- Creating a Command
- Requesting Command Output and Exit Code
The output can include both Standard Output and Standard Error streams.
Kind of like SSH but not.
What about Powershell Remoting
I'll save the details for another post but be aware there is a more modern protocol called the Powershell Remoting Protocol (PSRP). There is no cross platform implementation that I am aware of. It uses the same SOAP based wire protocol - MS-WSMV (Web Services Management Protocol Extensions for Windows Vista). I just love the "shout out" to Vista here.
PSRP is more feature rich but more difficult to implement. I've been playing with a ruby based partial implementation and will blog more details soon. You can run powershell commands with WinRM but you are shelling out to Powershell.exe from the traditional "fisher-price" Windows command shell.
Securing the transport
If you are using native WinRM on Windows (likely via Powershell Remoting) the most popular methods of authentication and encryption are (but are not limited to):
Negotiate
To quote the Windows Remote Management Glossary, Negotiate Authentication is defined:
A negotiated, single sign on type of authentication that is the Windows implementation of Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO). SPNEGO negotiation determines whether authentication is handled by Kerberos or NTLM. Kerberos is the preferred mechanism. Negotiate authentication on Windows-based systems is also called Windows Integrated Authentication.
Mmmmmmm. Lets all take a minute or two and think about just what this means and how it might guide our relationships and shape our perspectives on global social injustice.
Yeah its complex right? Anyways its also secure. Its much better than emailing your password to the contacts in your address book.
Kerberos
You know this Windows Remote Management Glossary is pretty darn smart. Lets just see what it has to say about Kerberos:
A method of mutual authentication between the client and server that uses encrypted keys. For computers running on a Windows-based operating system, the client account must be a domain account in the same domain as the server. When a client uses default credentials, Kerberos is the authentication method if the connection string is not one of the following: localhost, 127.0.0.1, or [::1].
Takeaways here are its "mutual", "encrypted" and importantly "for computers." Again, better than emailing your password to the contacts in your address book.
Basic Authentication
Secrets are hard. Not only technically but emotionally as well. There is no small effort involved hiding the truth. Well thank goodness Basic Authentication makes it easy to share our secrets easily. Ah sweet freedom. Here credentials are transmitted in plain text. I like to think of it as a "giving" protocol. Sure we all think our secrets are worth being kept but are they? Really? Just use Basic Authentication and you might find out.
SSL
This really isn't so much an authentication mechanism but it is a familiar means of securely transporting data from one point to another where the contents are only to be accessible to the sender and receiver. WinRM communication can use either HTTP or HTTPS (SSL). HTTP is the default but HTTPS provides an added layer of encryption.
One of the critical keys to securely using SSL is having a valid certificate issued by a reputable certification authority that serves to ensure that those on either side of the communication are who they say they are. Without this, for example using a non validated self signed certificate, you run the risk of a "Man in the Middle Attack". Not a nice man who offers to fix your tire, provide financial assistance or offer grievance counseling after the loss of a loved family member or pet. Rather a mean man who wants to take what you have and not give it back. He can do that because the authenticity of your certificate cannot be validated and therefore he can stand in the middle between you and the remote windows machine pretending to be that machine.
The dilemma of sane encryption using cross platform libraries
Some of the dilemmas I'll mention are shared in both native windows and cross platform libraries. For example, the friction of getting SSL up and running is pretty much the same on both sides. The key difference is that cross platform libraries may not (usually don't in fact) have access to all the authentication mechanisms listed above or getting them installed and configured is far less than clear.
Good news: SSL is Everywhere, Bad News: SSL is a pain to setup everywhere
Of course this statement is somewhat relative. There are those that are familiar with the basic rules of computer security and work with the knobs and levers of these mechanisms frequently enough that it is straight forward for them to setup. Even among the technically savvy, this is NOT the majority and its especially true in the world of WinRM vs. SSH. Not because SSH users are smarter, but because in Linux land, SSH is just so ubiquitous and pervasive its hard not to deal with regularly enough for it to be unfamiliar.
Here are some points to highlight the friction and pitfalls of SSL over WinRM:
- Its not on by default and there are several steps to set it up
- Without a "valid" certificate its still not secure (but better than Basic Authentication over HTTP) and valid certificates take effort to obtain.
- Bootstrapping problem: How do I get the valid certificate onto the remote machine before establishing a secure connection? There are several ways to do this depending on your cloud provider or image prep system but that assumes you have a cloud provider or an image prep system.
Kerberos: I cant tell you in a paragraph how to set it up and get it working
First just getting the right library can be the worst part - one that's compatible with your OS, architecture and the language runtime of your WinRM library. Most cross platform libraries support it but its less than trivial to get working.
Negotiate is likely not implemented
Well on ruby it is now, however, its the only non native Windows implementation I am aware of (which does not mean that there are not others). If it is implemented, and again - it is on Ruby, its secure and it "Just Works."
Basic auth is available everywhere, horrible everywhere and slightly painful to setup but easier than SSL
If you dig into almost all of the Readme files of the cross platform WinRM libraries, they will all tell you how to run horrible commands on your computer that put out the Welcome Mat for the bad guys. I've listed these in a few posts and for once I will not do so here.
Not only are running these commands a bad idea in general (certainly on production nodes) but again they represent a bootstrapping problem. Before you can successfully talk to the machine they have to be run. There are ways to accomplish this but again rely on cloud APIs or pre-baking images.
Next steps, caveats and how can we make this even better
Its totally awesome that NTLM/Negotiate authentication is now available as a cross platform option but it only lays down the foundation.
Its not the default and consuming applications need to "turn it on"
When using the WinRM gem, consumers must specify which authentication transport they want to use. There is no default. So today applications that specify ":plaintext, basic_auth: true" will continue to use basic authentication.
Chef and Test-Kitchen support coming very soon, and Vagrant support on the way
I am a developer at Chef and my PR for porting this into Knife-Windows is imminent. I will also be porting to Test-Kitchen's winrm transport and I plan to submit a PR to do the same for Vagrant.
Update: PRs to knife-windows, test-kitchen and vagrant are all submitted.
Still no support outside of Ruby
For instance Ansible (Python) and Packer (GO) - both very popular tools that help manage a data center have yet to have working Negotiate Auth implementations available.
The fact is its hard and tedious to implement something like an encryption specification solely from following a specification PDF. Further, there are other ways to get secure so its not a "show stopper." However as more become aware of the rubyntlm library, referencing such a library makes it much easier to implement.