Whitepaper – UltraVNC Encryption Overview

Introduction

Bozteck VENM Console can deploy UltraVNC with session level encryption enabled. This is recommended in network environments where you fear that your VNC session may be “sniffed” from the network and replayed back without your knowledge. The risk of someone successfully doing this on your LAN is very low as it does require a moderately high level of “hacker skill”. Some networks, however, need extra protection even against low risk attacks such as this.

This document will explain the principles behind the VENM supported UltraVNC encryption as well as how to deploy and enable it from the console. The skill level to use this information successful varies. If you simply accept the safe defaults, the skill level required is low. If you choose to make modifications such as creating a custom rc4.key file, the skill level required is much higher.

This document is for Admins who want to customize the encryption, get to know how it works at a lower level, or are having DSM errors when connecting to an encrypted server. Anyone else can skim through or read this document as they wish.

Basic Principles

vncviewer.exe –> {RC4.KEY} ===> (Network) ===> {RC4.KEY} ==> [msrc4plugin_noreg.dsm] –> winvnc.exe

There are two key files that are fundamental to UltraVNC session level encryption and they are “msrc4plugin_noreg.dsm” and “rc4.key”. These will be refered to as the DSM and KEY files respectively through the rest of this document.

When a vncviewer.exe file wants to talk to an UltraVNC server that is set to use encryption, it starts by looking for it’s copy of the KEY file. If found, it encrypts it’s connection attempt using the data in the KEY file. It, then, sends a connection request to the remote server using this encrypted mode of communication that would seem like garbage to anything else on the network.

On the other end, the server hears a connection being requested. Because it is also configured to require encryption, it consults it’s copy of the KEY file and uses the data in the KEY file to decrypt the communication request coming from the viewer. If the keys match up, the data is correctly deciphered and the two commence communication in this encrypted tunnel.

If the keys do not match, the connection attempt is rejected and the viewer presents an error to the user stating that possible reasons could include the failure to find a DSM file and a few other suggestions. The error message is vague but it pretty much just means that either:

• The remote server is not configured for encryption but you are sending an encrypted request
• The server is configured to require encryption but you are sending an unencrypted request
• Both ends are configured for encryption but the KEY files do not match

More About the KEY and DSM Files (Server End)

The DSM file is a plug-in to the winvnc.exe service file. For reference, the winvnc.exe is the actual server executable for UltraVNC that is deployed to the remote computers and installed as a service. The DSM file must reside in the same folder as the winvnc.exe file. Don’t worry too much about that because the Bozteck VNC Deployment Tool does that work for you. When the DSM file is in the same folder as the winvnc.exe file and the configuration for the server has been properly set, the server will attempt to encrypt communication for any client that attempts to connect to it.

The KEY file also must reside in the same folder as the winvnc.exe file. The Deployment Tool also make sure that this happens as long as you do not modify things too much. If you simply tell the Profile Editor to enable encryption, it takes the KEY file that is located in c:fastpushvnc7ultra and copies it to your Deployment Profile’s folder for easy deployment every time that you push that profile out to a computer.

Because this KEY file must match the KEY file on the Viewer end, it is best practice to only use one KEY file for your entire network. You do not gain sufficient security advantage by using multiple KEY files to make up for the headache required to keep them all straight and matched up. If you choose to generate a new KEY file, you must push that new KEY file out to each and every computer on your network that has encryption enabled to ensure proper connectivity.

By default, the UltraVNC Server is installed to either “C:Program Filesorlvnc” or “C:Program Filesultravnc” depending on the version of UltraVNC you are using. If you choose to distribute the KEY file using some method other than a VNC Push from VENM, it must in somehow make it’s way into that path on the remote computer end. Again, the VNC Push from VENM does this for you.

More About the KEY and DSM Files (Viewer End)

The KEY file is required to be in the same folder as the vncviewer.exe file. This KEY file must be an exact copy of the KEY file that is located at the server end as described above. By default, VENM uses the vncviewer.exe located in “C:Program FilesBozteckVNCScan Console .NET” or whatever location you have installed VENM to.

You have the ability to customize the location of your vncviewer.exe file using either the main preferences (Support Files section), group properties (VNC Settings section), or the computer properties (VNC Settings section). If you choose to modify the location of the vncviewer.exe file, you need to make sure that a copy of your RC4.KEY file is moved with it.

Creating Custom KEYs

The Profile Editor has a button in the “Customs” section that allows you to generate a new key. This is an advanced level function and should never really need to be done in normal circumstances. To my knowledge, there has never been a tool released that allows someone to decrypt an encrypted UltraVNC stream for later playback. You should only need to generate a custom key if this is ever done. The KEY file does not control who can access you server. It only controls the way that the session is encrypted. You will still need to authenticate with the remote server using your password even if the KEY files match up.

If you do choose to create a custom KEY file, pressing this button is the easiest way to make sure that everything is placed in the correct locations for easy deployment. You may need to manually copy the new KEY file to the location where your vncviewer.exe resides, however. The newly generated key is always placed in c:fastpushvnc7ultra.

Conclusion

I hope that this was a good in-depth overview of how the UltraVNC encryption works and how it interacts within the Bozteck VNCScan Enterprise Network Manager (VENM).