# OpenVPN Padlock



## cdm710 (Oct 1, 2009)

I'm trying to get routed OpenVPN to work with a couple of VIA C7 machines on FreeBSD 6.2 so I can utilize the Padlock engine to accelerate OpenVPN. So far it's not working for some reason and I'm not sure why. I have a server machine with a single nic behind a FreeBSD 6.2 IPF firewall and a single nic FreeBSD 6.2 client machine and I've set it up so the subnet behind the client can access the subnet behind the server. This all works fine except for the fact that the Padlock acceleration isn't working. I'm trying to use 'cipher AES-128-CBC' on both the server and client and I also put 'engine padlock' at the end of both config files. 

When I run an openssl speed test using the padlock engine it's significantly faster than the cryptodev engine, so it seems like padlock is working, but it's not with OpenVPN because I notice no speed difference at all. When I run 'openssl engine padlock' it returns '(padlock) VIA PadLock (no-RNG, ACE)'. I have crypto.ko, cryptodev.ko, and padlock.ko loading at startup. When the system starts, it displays the following errors: Padlock: 'No ACE support.' and 'module_register_init: MOD_LOAD (padlock, 0xc34cd57c, 0) error 22'. Another odd thing is that when I run 'dmesg | grep CPU' it returns this:

```
CPU: VIA/IDT Unknown (1999.59-MHz 686-class CPU)
cpu0: <ACPI CPU> on acpi0
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
CPU: VIA/IDT Unknown (1999.60-MHz 686-class CPU)
cpu0: <ACPI CPU> on acpi0
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
```
It seems that the OS isn't properly recognizing the CPU, which might be part of the problem, but I'm not sure what to do about that. How do you guys think I should proceed? Try FreeBSD 7.2?


----------



## cdm710 (Oct 2, 2009)

I'm using the version of OpenSSL that came with 6.2, which is 0.9.7e-p1. This version doesn't appear to be working properly with Padlock. I did testing using scp and got the following results:


```
#scp -c aes128-cbc myfile.dat root@localhost:/dev/null
myfile.dat                                    100%  100MB   5.6MB/s   00:18

# scp -c blowfish myfile.dat root@localhost:/dev/null
myfile.dat                                    100%  100MB   8.3MB/s   00:12
```

Clearly Padlock acceleration is not working and I suspect OpenSSL is the problem. I used portsnap to update the ports collection and tried updating OpenSSL. It wasn't recognizing the upgrade attempt it kept saying I was using 0.9.7e-p1, so I did some digging and found this page:

http://lists.freebsd.org/pipermail/freebsd-questions/2008-November/186441.html

After editing /root/.cshrc like the above link suggests, the OS recognizes the new version of OpenSSL, version 0.9.8k, but now when I run 'openssl engine' it only lists cryptodev and dynamic. Are the padlock kernel modules I need compatible with this version of OpenSSL? How do I get it to recognize where they are? Am I on the right track?


----------



## cdm710 (Oct 2, 2009)

I had a spare HD sitting around so I decided to install FreeBSD 7.2 on it to see how the machine I've been having difficulties with would behave. 7.2 seems more promising, but it still seems like it isn't working properly. Here are the results of dmesg that lead me to believe I'm getting closer:


```
# dmesg |grep CPU
CPU: VIA C7 Processor 2000MHz (1999.59-MHz 686-class CPU)
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0

# dmesg |grep Padlock
  VIA Padlock Features=0xffcc<RNG,AES,AES-CTR,SHA1,SHA256,RSA>

# dmesg |grep padlock
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
```

I have the following kernel modules loaded:


```
# kldstat
Id Refs Address    Size     Name
 1   12 0xc0400000 9fab28   kernel
 2    3 0xc0dfb000 25ff8    crypto.ko
 3    3 0xc0e21000 ab40     zlib.ko
 4    1 0xc0e2c000 442c     cryptodev.ko
 5    1 0xc0e31000 4b64     padlock.ko
 6    1 0xc0e36000 6a45c    acpi.ko
 7    1 0xc312a000 e000     ipfw.ko
 8    1 0xc3177000 22000    linux.ko
```

My scp results are about the same as before:


```
# scp -c aes128-cbc myfile.dat root@localhost:/dev/null
myfile.dat                                    100%  100MB   6.3MB/s   00:16

# scp -c blowfish myfile.dat root@localhost:/dev/null
myfile.dat                                    100%  100MB   9.1MB/s   00:11
```

What am I doing wrong here?


----------



## honk (Oct 2, 2009)

Is your scp transfer rate slower if padlock.ko is not loaded?


----------



## plamaiziere (Oct 2, 2009)

cdm710 said:
			
		

> Try FreeBSD 7.2?



Yes, see:
http://www.freshbsd.org/?branch=&project=freebsd&committer=&module=src&q=padlock


----------



## cdm710 (Oct 3, 2009)

honk said:
			
		

> Is your scp transfer rate slower if padlock.ko is not loaded?



No, it's the same speed


----------



## cdm710 (Oct 3, 2009)

plamaiziere said:
			
		

> Yes, see:
> http://www.freshbsd.org/?branch=&project=freebsd&committer=&module=src&q=padlock



I'm running the i386 version of 7.2


----------



## plamaiziere (Oct 3, 2009)

cdm710 said:
			
		

> I'm running the i386 version of 7.2



And with 7.2, does padlock(4) recognize the ACE engine?

You have two ways to use the ACE engine : the first is with openssl and the padlock engine. In this case openssl uses the special instructions set for padlock. With the driver padlock(4), you must use the engine cryptodev. The best is to use the openssl padlock engine because there is no need to go in the kernel.

If you want to check both, use openssl to encrypt or decrypt a big file 

Something like :

```
openssl enc -e -aes-128-cbc -k toto -in abigfile -out /dev/null -nosalt [-engine padlock | -engine cryptodev | nothing]
```
If you see a large difference in speed, ACE is working.

Then check with openVPN, on freebsd > 7 there is a bug and openssl does not use by default the cryptodev driver. You must patch openssl, I use an old patch and I don't know if it still applies :


```
--- eng_cryptodev.c.orig	2008-02-05 18:10:31.000000000 +0000
+++ eng_cryptodev.c	2008-06-14 18:25:36.175353823 +0100
@@ -1127,6 +1127,7 @@
 	}
 
 	ENGINE_add(engine);
+	ENGINE_set_default_ciphers(engine);
 	ENGINE_free(engine);
 	ERR_clear_error();
 }
```


----------



## cdm710 (Oct 4, 2009)

plamaiziere said:
			
		

> And with 7.2, does padlock(4) recognize the ACE engine?



Yes, it recognizes it. When I type 'openssl engine' padlock with ACE support is in the list.



			
				plamaiziere said:
			
		

> If you want to check both, use openssl to encrypt or decrypt a big file
> 
> Something like :
> Code:
> ...



It's significantly faster when I use the padlock engine. Cryptodev and nothing are about the same speed.

Forgive my ignorance, but what exactly do I apply the suggested patch to? Thanks for the help, I feel like I'm getting somewhere.


----------



## plamaiziere (Oct 5, 2009)

cdm710 said:
			
		

> Yes, it recognizes it. When I type 'openssl engine' padlock with ACE support is in the list.
> 
> It's significantly faster when I use the padlock engine. Cryptodev and nothing are about the same speed.
> 
> Forgive my ignorance, but what exactly do I apply the suggested patch to? Thanks for the help, I feel like I'm getting somewhere.



Ok so you don't need to use the cryptodev engine. You just need to tell to openVPN to use openssl with the engine padlock. I don't know how you can achieve this.

The patch I posted was for the cryptodev engine.


----------



## cdm710 (Oct 6, 2009)

Ok so I put FreeBSD 7.2 on both machines I'm working on and I reran this test: 

```
openssl enc -e -aes-128-cbc -k toto -in abigfile -out /dev/null -nosalt [-engine padlock | -engine cryptodev | nothing]
```
Now padlock and cryptodev are about the same speed on both machines, both significantly faster than 'nothing'. Not sure why cryptodev was slower before... I tested OpenVPN using both 'engine padlock' and 'engine cryptodev' and it's still not working. Both of those engines show up as options when I run 'openvpn --show-engines'. 

So how do I apply that cryptodev patch that was posted?


----------



## cdm710 (Oct 6, 2009)

Ok I figured out how to apply the patch (I think) and it's still not working. I ran '/usr/src/patch < cryptodev_patch' where cryptodev_patch contained the patch code you provided. Tried OpenVPN with both 'engine padlock' and 'engine cryptodev' and no speed boost still. Frustrating...


----------



## plamaiziere (Oct 6, 2009)

cdm710 said:
			
		

> Ok I figured out how to apply the patch (I think) and it's still not working. I ran '/usr/src/patch < cryptodev_patch' where cryptodev_patch contained the patch code you provided. Tried OpenVPN with both 'engine padlock' and 'engine cryptodev' and no speed boost still. Frustrating...



I've not tested this patch with OpenVPN but it worked for sure with OpenSSH. 

Which cyphers are you using with OpenVPN? padlock(4) will be used only if it is able to handle *all* the cyphers.

You can use the tools located in /usr/src/tools/tools/crypto to see what happens with the crypto framework (cryptostats tool and cryptotest tool)


----------



## cdm710 (Oct 6, 2009)

plamaiziere said:
			
		

> I've not tested this patch with OpenVPN but it worked for sure with OpenSSH.
> 
> Which cyphers are you using with OpenVPN? padlock(4) will be used only if it is able to handle *all* the cyphers.
> 
> You can use the tools located in /usr/src/tools/tools/crypto to see what happens with the crypto framework (cryptostats tool and cryptotest tool)



I'm trying to use aes-128-cbc with openvpn. I managed to install the tools you referred to, but I'm not sure what to do with them. Here's what happens when I run cryptostats:

```
# /usr/local/bin/cryptostats
0 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations
```
I tried running that while doing an openssl speed test using the padlock engine, cryptodev engine, and the default software engine and I got the same results. I guess that's not how you're supposed to use that command, but I can't seem to figure out what to do with it...


----------



## honk (Oct 6, 2009)

I played around on my 7.2-RELEASE-p4 VIA C7 box. After updating from OpenSSL 0.9.8e to OpenSSL 0.9.8k the transfer rate with scp increased from 3MBytes/sec to 10MBytes/sec 

cheers,
honk


----------



## cdm710 (Oct 7, 2009)

honk said:
			
		

> I played around on my 7.2-RELEASE-p4 VIA C7 box. After updating from OpenSSL 0.9.8e to OpenSSL 0.9.8k the transfer rate with scp increased from 3MBytes/sec to 10MBytes/sec
> 
> cheers,
> honk


I really hate to ask this, but how did you update to 0.9.8k? Updating from the ports tree isn't working and I can't seem to figure out how to do it. I'll keep looking, but I would really appreciate some help. Thanks.


----------



## DutchDaemon (Oct 7, 2009)

Why isn't 'updating from ports' working? Just call /usr/local/bin/openssl, or put /usr/local earlier in your $PATH, or move /usr/bin/openssl out of the way, making it a symlink to /usr/local/bin/openssl. Or switch to 8.0 (which has 0.98k in the base).


----------



## cdm710 (Oct 7, 2009)

DutchDaemon said:
			
		

> Why isn't 'updating from ports' working? Just call /usr/local/bin/openssl, or put /usr/local earlier in your $PATH, or move /usr/bin/openssl out of the way, making it a symlink to /usr/local/bin/openssl. Or switch to 8.0 (which has 0.98k in the base).


I created a symlink as you suggested and now the padlock engine isn't listed anymore when I type 'openssl engine', so that doesn't quite work as intended. I also tried putting /usr/local earlier in $PATH in a couple of different places in front of /usr/bin which also didn't work because openssl still reported that it was version 0.9.8e. I'm assuming you were talking about modifying path in /root/.cshrc right?


----------



## DutchDaemon (Oct 7, 2009)

Are /etc/ssl/openssl.cnf and /usr/local/openssl/openssl.cnf 'in agreement'?
[cmd=]diff /etc/ssl/openssl.cnf /usr/local/openssl/openssl.cnf[/cmd] or even [cmd=]diff /etc/ssl/openssl.cnf /usr/local/openssl/openssl.cnf.sample[/cmd] if you haven't created it yet ..)


----------



## cdm710 (Oct 7, 2009)

DutchDaemon said:
			
		

> Are /etc/ssl/openssl.cnf and /usr/local/openssl/openssl.cnf 'in agreement'?
> [cmd=][/cmd] or even [cmd=]diff /etc/ssl/openssl.cnf /usr/local/openssl/openssl.cnf.sample[/cmd] if you haven't created it yet ..)


When I copy /etc/ssl/openssl.cnf to /usr/local/openssl/openssl.cnf and run 'openssl engine' cryptodev and dynamic show up. When I run 'diff /etc/ssl/openssl.cnf /usr/local/openssl/openssl.cnf.sample' I get the following:

```
1d0
< # $FreeBSD: src/crypto/openssl/apps/openssl.cnf,v 1.8.8.1 2009/04/15 03:14:26 kensmith Exp $
16,26d14
< #enable padlock engine by default:
< #openssl_conf = openssl_def
< #[openssl_def]
< #engines = openssl_engines
< #[openssl_engines]
< #padlock = padlock_engine
< #[padlock_engine]
< #default_algorithms = ALL
< #end of crazy configfile syntax
<
<
49c37
< dir           = /usr/local/etc/openvpn                # Where everything is kept
---
> dir           = ./demoCA              # Where everything is kept
57c45
< certificate   = $dir/certs/cacert.pem         # The CA certificate
---
> certificate   = $dir/cacert.pem       # The CA certificate
80c68
< default_days  = 3650                  # how long to certify for
---
> default_days  = 365                   # how long to certify for
137c125
< countryName_default           = US
---
> countryName_default           = AU
142c130
< stateOrProvinceName_default   = Ohio
---
> stateOrProvinceName_default   = Some-State
145d132
< localityName_default          = Columbus
148c135
< 0.organizationName_default    = xxxxx
---
> 0.organizationName_default    = Internet Widgits Pty Ltd
158d144
< commonName_default            = Enter User's Name
162d147
< emailAddress_default          = username@xxxxx
196c181
< nsCertType = client, email, objsign
---
> # nsCertType = client, email, objsign
206c191
< authorityKeyIdentifier=keyid,issuer:always
---
> authorityKeyIdentifier=keyid,issuer
```
If I just copy openssl.cnf.sample over openssl.cnf in /usr/local/openssl and run 'openssl engine' cryptodev and dynamic are still the only engines that are recognized.


----------



## DutchDaemon (Oct 7, 2009)

What I'm trying to say is that if you want to use openssl from _ports_, you'll have to use /usr/local/openssl/openssl.cnf. So if you want Padlock with openssl from ports, you'll have to put the padlock stuff in the correct config file. Note that you had Padlock commented out in the diff you posted above, so I guess that's what prevents it from showing up now.


----------



## cdm710 (Oct 8, 2009)

DutchDaemon said:
			
		

> What I'm trying to say is that if you want to use openssl from _ports_, you'll have to use /usr/local/openssl/openssl.cnf. So if you want Padlock with openssl from ports, you'll have to put the padlock stuff in the correct config file. Note that you had Padlock commented out in the diff you posted above, so I guess that's what prevents it from showing up now.


The padlock code in the openssl.cnf file just tells openssl to use the padlock engine by default. Even if I uncomment it 'openssl engine' doesn't show padlock as being supported. It appears that for some reason it's not recognizing the kernel module and I'm not sure why. I tried unloading the padlock kernel module and reloading it, but that didn't do anything.


----------



## cdm710 (Oct 8, 2009)

honk said:
			
		

> I played around on my 7.2-RELEASE-p4 VIA C7 box. After updating from OpenSSL 0.9.8e to OpenSSL 0.9.8k the transfer rate with scp increased from 3MBytes/sec to 10MBytes/sec
> 
> cheers,
> honk


Would you mind explaining the process you used to upgrade from 0.9.8e to 0.9.8k? Clearly I did something different than you.


----------



## cdm710 (Oct 8, 2009)

Ok, so I uncommented the following in /usr/local/openssl/openssl.cnf and restarted the system:

```
#enable padlock engine by default:
openssl_conf = openssl_def
[openssl_def]
engines = openssl_engines
[openssl_engines]
padlock = padlock_engine
[padlock_engine]
default_algorithms = ALL
```
Now when I run 'openssl engine' I get the following error:

```
Error configuring OpenSSL
2345:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:162:filename(/usr/local/lib/engines
/libpadlock.so): Cannot open "/usr/local/lib/engines/libpadlock.so"
2345:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:244:
2345:error:260B6084:engine routines:DYNAMIC_LOAD:dso not found:eng_dyn.c:450:
2345:error:2606A074:engine routines:ENGINE_by_id:no such engine:eng_list.c:415:id=padlock
2345:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=openssl_engines, 
retcode=-1
```
What's odd is that I don't appear to have libpadlock.so anywhere on my system. Is this an OpenSSL bug or something? Another thing that's different is when I run dmesg |grep padlock the following is returned:

```
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
padlock0: detached
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
padlock0: detached
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
padlock0: detached
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
padlock0: <AES-CBC,SHA1,SHA256> on motherboard
```
I'm assuming this is happening because OpenSSL can't load the padlock engine.


----------



## cdm710 (Oct 8, 2009)

I found this page: http://www.logix.cz/michal/devel/padlock/#openssl098-static
The patches appear to have been taken from Fedora 5, so I'm not sure they're suitable for FreeBSD. Also, if I apply the patches and recompile, will that recompile version 0.9.8e which is in the base distribution, or version 0.9.8k which I installed from the ports collection?


----------



## honk (Oct 8, 2009)

From my understanding userland applications use cryptodev to access hardware crypto accelerators. Therefore I don't cared that the "openssl engine" command did not show the padlock engine (but cryptodev). Your dmesg output (the padlock0 detached messages) might appear because you kldload/kldunload the padlock.ko module multiple times?

I updated OpenSSL the following way (shame on me):

```
cd /usr/ports/security/openssl
make PREFIX=/ install
```

The procedure might be or is completely wrong, my intention was just to replace the OpenSSL installation which comes with the base distribution. I did not want to have >>multiple<< OpenSSL versions on my box. I guess it's key to have the shared libs updated under /lib (not /usr/local/lib) so that OpenSSH (without recompiling?!) can use it.

As a test I booted a fresh 7.2-RELEASE, run the scp tests again (gave me 3MBytes/sec), after that I just copied the following libs from my other box (the ones after my dirty OpenSSL update):


```
# ll /lib | grep crypto
-rw-r--r--  1 root  wheel  2380268  6 Okt 23:26 libcrypto.a
-r--r--r--  1 root  wheel  1536226  6 Okt 23:26 libcrypto.so.5
```

With that libs, scp suddenly gave me 4,5MBytes/sec. After a "kldload cryptodev" it was 10MBytes/sec and with top I could see that the CPU utilization moved from userspace to kernel space.

I would be happy if somebody could explain the canonical way to >>replace<< the OpenSSL version of the base distribution with a update from ports. Means avoid having multiple versions of OpenSSL.

On the other hand, I'm waiting for the release of 8.0, which should solve these problems.

cheers,
honk


----------



## cdm710 (Oct 9, 2009)

Thanks for explaining the method you used to upgrade OpenSSL. I'm using 0.9.8k on both systems now. Using your method I was able to successfully increase my scp rate from 5.9 MB/s to 14.7 MB/s. When I look at the 'top' output I can clearly see that the CPU utilization has shifted from userspace to kernel space like you said and I can see scp and ssh appear and see that they're being used. When I run openssl speed tests pretty much the same is true except that I see the process 'openssl' show up and the CPU usage under 'system' is ~100%.  

Unfortunately, none of this seems to have carried over to OpenVPN. OpenSSL now indicates that it supports the cryptodev and dynamic engines and so does OpenVPN, so I tell OpenVPN to use 'engine cryptodev' in my config files. When I copy files between samba shares on the servers through the VPN, I'm still not seeing any speed increase and when I look at 'top' while I'm transferring I'm not really seeing much of anything; no openssl/openvpn process ~100% CPU usage under 'system', pretty much nothing even indicates OpenVPN is even being used. The 'top' output indicates that 'user' and 'system' CPU usage are each ~1%.


----------



## cdm710 (Oct 12, 2009)

Ok, another thing I've noticed that doesn't seem to be working is the sha1/sha256 acceleration. Here's the results of my test:


```
# openssl speed -evp sha1 -engine cryptodev
engine "cryptodev" set.
To get the most accurate results, try to run this
program when this computer is idle.
Doing sha1 for 3s on 16 size blocks: 1477447 sha1's in 2.99s
Doing sha1 for 3s on 64 size blocks: 1066112 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 573495 sha1's in 2.99s
Doing sha1 for 3s on 1024 size blocks: 201385 sha1's in 2.99s
Doing sha1 for 3s on 8192 size blocks: 28559 sha1's in 2.99s
OpenSSL 0.9.8k 25 Mar 2009
built on: Fri Oct  9 09:25:41 EDT 2009
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -pthread -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIOS -O3 -fomit-
frame-pointer -Wall -O2 -fno-strict-aliasing -pipe -DOPENSSL_BN_ASM_PART_WORDS -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: USE_TOD HZ=128 [sysconf value]
timing function used: getrusage
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
sha1              7898.16k    22777.46k    49032.53k    68888.21k    78161.00k
```
Clearly not being accelerated. Could this be why OpenVPN isn't any faster? Any ideas on how to enable sha1/sha256 acceleration?


----------



## cdm710 (Oct 13, 2009)

Thanks for the help guys, but I can't waste anymore time on this. As much as I hate to say it, I don't think FreeBSD is the right choice for this at the moment. I love FreeBSD, but it seems like Linux support for Padlock is more mature at this point so I'm switching to Debian.


----------



## cdm710 (Oct 14, 2009)

Worked pretty much immediately with Debian 5.0 :e


----------

