# Security Protocol Processor (hardware encryption)



## do_more_secrets (Apr 4, 2013)

Did anyone get this card to run on FreeBSD 9.1?

http://www.ebay.de/itm/180777328545

It's using a Broadcom BCM5862 chip for hardware accelerated encryption. Especially the AES-CBC support would be very interesting.


----------



## SirDice (Apr 4, 2013)

There are a few Broadcom chipsets mentioned in ubsec(4) but I don't see this particular one. The man page may not reflect all supported chipsets though.


----------



## wblock@ (Apr 4, 2013)

Some mention of it here: http://lists.freebsd.org/pipermail/freebsd-hackers/2006-December/018927.html

This old mailing list post suggests that the BCM5862 is a BCM5825, which ubsec(4) says it supports.


----------



## do_more_secrets (Apr 4, 2013)

If I understand the man pages for ubsec right....


> Broadcom BCM5823  A BCM5822 with AES capability.
> Broadcom BCM5825  Faster version of the BCM5823.
> ....
> The AES capability of the BCM5823 is not yet supported; it is awaiting public disclosure
> of programming information from Broadcom.


... then it's without support for the AES-part


----------



## Terry_Kennedy (Apr 16, 2013)

do_more_secrets said:
			
		

> Did anyone get this card to run on FreeBSD 9.1?
> 
> http://www.ebay.de/itm/180777328545
> 
> It's using a Broadcom BCM5862 chip for hardware accelerated encryption. Especially the AES-CBC support would be very interesting.


I figured I'd look into this, and the card just arrived. It is not detected in FreeBSD 8.4-RC1, and a quick look in SVN doesn't show any added hardware support in HEAD (10.x).

I whipped up some quick patches so FreeBSD would at least detect the card. I haven't done any testing yet to see if it can actually be used as a crypto accelerator. Here are the diffs:

```
*** ubsec.c_orig	Sun Mar 24 22:03:36 2013
--- ubsec.c	Mon Apr 15 18:04:48 2013
***************
*** 217,223 ****
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5821 ||
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5822 ||
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5823 ||
! 	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5825
  	     ))
  		return (BUS_PROBE_DEFAULT);
  	return (ENXIO);
--- 217,224 ----
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5821 ||
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5822 ||
  	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5823 ||
! 	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5825 ||
! 	     pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5862
  	     ))
  		return (BUS_PROBE_DEFAULT);
  	return (ENXIO);
***************
*** 238,243 ****
--- 239,245 ----
  		case PCI_PRODUCT_BROADCOM_5822:	return "Broadcom 5822";
  		case PCI_PRODUCT_BROADCOM_5823:	return "Broadcom 5823";
  		case PCI_PRODUCT_BROADCOM_5825:	return "Broadcom 5825";
+ 		case PCI_PRODUCT_BROADCOM_5862:	return "Broadcom 5862";
  		}
  		return "Broadcom unknown-part";
  	case PCI_VENDOR_BLUESTEEL:
***************
*** 300,306 ****
  	     (pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5821 ||
  	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5822 ||
  	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5823 ||
! 	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5825)) ||
  	    (pci_get_vendor(dev) == PCI_VENDOR_SUN &&
  	     (pci_get_device(dev) == PCI_PRODUCT_SUN_SCA1K ||
  	      pci_get_device(dev) == PCI_PRODUCT_SUN_5821))) {
--- 302,309 ----
  	     (pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5821 ||
  	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5822 ||
  	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5823 ||
! 	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5825 ||
! 	      pci_get_device(dev) == PCI_PRODUCT_BROADCOM_5862)) ||
  	    (pci_get_vendor(dev) == PCI_VENDOR_SUN &&
  	     (pci_get_device(dev) == PCI_PRODUCT_SUN_SCA1K ||
  	      pci_get_device(dev) == PCI_PRODUCT_SUN_5821))) {

*** ubsecreg.h_orig	Sun Mar 24 22:03:36 2013
--- ubsecreg.h	Mon Apr 15 18:01:45 2013
***************
*** 63,68 ****
--- 63,69 ----
  #define	PCI_PRODUCT_BROADCOM_5822	0x5822		/* 5822 */
  #define	PCI_PRODUCT_BROADCOM_5823	0x5823		/* 5823 */
  #define	PCI_PRODUCT_BROADCOM_5825	0x5825		/* 5825 */
+ #define	PCI_PRODUCT_BROADCOM_5862	0x5862		/* 5862 */
  
  /* Sun Microsystems */
  #define PCI_PRODUCT_SUN_5821		0x5454		/* Crypto 5821 */
```
Note that this may not be the correct way to handle the 5862 - I'm going by the Broadcom product brief which says it is upwardly compatible from the 5825.

Here's the [cmd=""]dmesg[/cmd] output when detecting the card:

```
cryptosoft0: <software crypto> on motherboard
ubsec0 mem 0xdfdf0000-0xdfdfffff irq 16 at device 0.0 on pci8
ubsec0: [ITHREAD]
ubsec0: Broadcom 5862
```


----------



## Terry_Kennedy (Apr 17, 2013)

Terry_Kennedy said:
			
		

> Note that this may not be the correct way to handle the 5862 - I'm going by the Broadcom product brief which says it is upwardly compatible from the 5825.


It doesn't seem like this is the correct way to handle the 5862. security/openssl doesn't benchmark any faster with -engine cryptodev, and both [cmd=""]cryptostats[/cmd] and [cmd=""]ubsecstats[/cmd] report 0 operations even after specifying the engine.

I booted an OpenBSD LiveCD and it did detect the card, as well as listing a load of crypto algorithms it supports. However, I couldn't test the OpenSSL -engine option as the LiveCD had a sharable crypto library in the wrong location. I also couldn't find where OpenBSD keeps the ubsec driver in their source tree (quick look via CVSweb).

If there's any real interest in this, I can look into it further and see what's needed to get OpenBSD's 5862 support into FreeBSD. If there isn't any interest, I'll put the card back in the stockroom. [Note that the company that made the Dell card in question no longer produces cards based on Broadcom security processors and the information on this particular card has been removed from their web site, so it may be too late for most people to care anyway.]


----------



## do_more_secrets (Apr 26, 2013)

@Terry_Kennedy, thank you very much. But I'm looking for a card that is able to work with FreeBSD and NAS4Free right out of the box. 



> Note that the company that made the Dell card in question no longer produces cards based on Broadcom security processors



In this case I have to find a more recent card. In Germany this card is hard to get and expensive and if it doesn't benchmark any faster..


----------



## wblock@ (Apr 26, 2013)

Even if encryption is not any faster, the system may be faster overall from reduced load on the CPU.


----------



## do_more_secrets (Apr 28, 2013)

*E*specially on my NAS4Free encryption is the only purpose. I'm a little bit astonished that questions about encryption and related subjects don't receive more attention. I think encryption is vital these days.

Does anyone of you know a card that works out of the box?


----------



## patpro (Apr 28, 2013)

Soekris makes some hardware that is fully supported on Open/FreeBSD: http://soekris.com/products/vpn14x1.html

I've never tested those cards, but I'm pretty confident you'll find some users in the Soekris community that run them on various FreeBSD flavors.


----------



## Terry_Kennedy (Apr 30, 2013)

patpro said:
			
		

> Soekris makes some hardware that is fully supported on Open/FreeBSD: http://soekris.com/products/vpn14x1.html
> 
> I've never tested those cards, but I'm pretty confident you'll find some users in the Soekris community that run them on various FreeBSD flavors.


Unless Broadcom is using a different metric, the card that started this post off is a *lot* faster than the Soekris card. Broadcom claims 2 GBit/sec throughput and 15,000+ RSA operations/sec, while the Soekris card claims 250 Mbit/sec throughput and 24 to 70 RSA connections/sec. I would think a modern performance CPU would outperform this Soekris card by a large margin (but that's just a guess).

More importantly, the original poster was asking about a PCI Express card while the Soekris is PCI (non-Express, different slot).


----------



## do_more_secrets (Apr 30, 2013)

I'm using a HP N40L ProLiant Server and I got 490 MBit/sec when uploading a file. The file is stored in a 128Bit AES-CBC container. When uploading a file the CPU runs at 100%. The 250 MBit/sec of the Soekris card is not a real step forward


----------



## patpro (May 1, 2013)

Well, I do agree, Soekris hardware is not powerful. It's meant to be used in very small form factor fanless appliance (think PicoBSD) were it can offload crypto from the small CPU.


----------



## Terry_Kennedy (May 10, 2013)

do_more_secrets said:
			
		

> I'm using a HP N40L ProLiant Server and I got 490 MBit/sec when uploading a file. The file is stored in a 128Bit AES-CBC container. When uploading a file the CPU runs at 100%. The 250 MBit/sec of the Soekris card is not a real step forward


Well, for no particular reason I started working on the ubsec(4) driver, and I now have it to the point of recognizing and using the Broadcom 5862. There is still quite some way to go, as "unusual" input (like giant buffers) passed to the device cause it to hang, sshd hangs in crydev state, and so on. However, take a look at this:


```
[cmd=root@poolx:~ #]kldload crypto[/cmd]
[cmd=root@poolx:~ #]kldload cryptodev[/cmd]
[cmd=root@poolx:~ #]kldload ubsec[/cmd]
cryptosoft0: <software crypto> on motherboard
ubsec0 mem 0xdf6f0000-0xdf6fffff irq 40 at device 0.0 on pci9
ubsec0: [ITHREAD]
ubsec0: Broadcom 5862 (3DES MD5 SHA1 AES)
[cmd=root@poolx:~ #]cryptotest -z 1024 16384[/cmd]
   0.319 sec,    2048    des crypts,   16384 bytes, 105327294 byte/sec,   803.6 Mb/sec
   0.319 sec,    2048   3des crypts,   16384 bytes, 105305808 byte/sec,   803.4 Mb/sec
   0.117 sec,    1024    md5 crypts,   16384 bytes, 142800616 byte/sec,  1089.5 Mb/sec
   0.099 sec,    1024   sha1 crypts,   16384 bytes, 170023268 byte/sec,  1297.2 Mb/sec
[cmd=root@poolx:~ #]ubsecstats[/cmd]
input 10911744 bytes 6144 packets
output 10911744 bytes 6144 packets
invalid 0 badsession 0 badflags 0
nodesc 0 badalg 0 nomem 0 queuefull 0
dmaerr 0 mcrerr 0 nodmafree 0
lenmismatch 0 skipmisatch 0 iovmisalined 0
noirq 0 unaligned 0 nomap 0 noload 0 nomcl 0
totbatch 0 maxbatch 0
maxqueue 1 maxqchip 1 mcr1full 0
rng 1 modexp 0 moexpcrt 0
[cmd=root@poolx:~ #]time cryptotest -z 102400 16384[/cmd]
  32.441 sec,  204800    des crypts,   16384 bytes, 103430868 byte/sec,   789.1 Mb/sec
  32.440 sec,  204800   3des crypts,   16384 bytes, 103434968 byte/sec,   789.1 Mb/sec
  12.108 sec,  102400    md5 crypts,   16384 bytes, 138564141 byte/sec,  1057.2 Mb/sec
  10.146 sec,  102400   sha1 crypts,   16384 bytes, 165357853 byte/sec,  1261.6 Mb/sec
0.300u 10.940s 1:27.13 12.9%    16+1537k 0+0io 0pf+0w
[cmd=root@poolx:~ #][/cmd]
```


----------



## tanked (May 10, 2013)

I'm constantly impressed with open source projects and developers, a message is posted on a forum enquiring about hardware that doesn't currently work and someone helpfully replies with patches to get it working, absolutely remarkable. Try getting the same result with a post to the Microsoft technet forums.


----------



## do_more_secrets (May 10, 2013)

I'm really impressed that you simply adapt a driver. The data throughput is no less impressive. (I miss the values for AES in your listing). I have a really bad conscience that I have looked at a card with Cavium CPU in the meantime. (Silicom PESC62).

However, I still didn't buy me a card because nearly no one sends these cards to Germany.


----------



## Terry_Kennedy (May 11, 2013)

do_more_secrets said:
			
		

> I'm really impressed that you simply adapt a driver. The data throughput is no less impressive. (I miss the values for AES in your listing).


The existing FreeBSD ubsec(4) driver (which is what I started with) doesn't support AES on newer (relatively speaking) cards. There's additional support for AES in the OpenBSD driver, and if I continue working on this project I'll merge that in.


> I have a really bad conscience that I have looked at a card with Cavium CPU in the meantime. (Silicom PESC62).
> 
> However, I still didn't buy me a card because nearly no one sends these cards to Germany.


I don't blame you for looking at a card that says it has a (presumably current and working) FreeBSD driver.

Unfortunately, the CPU in the system you're using doesn't support aesni(4). If it did, that would be the perfect solution to your AES encryption requirements.


----------



## Terry_Kennedy (May 12, 2013)

Terry_Kennedy said:
			
		

> Unfortunately, the CPU in the system you're using doesn't support aesni(4). If it did, that would be the perfect solution to your AES encryption requirements.


I re-ran the same tests from my reply above on the same hardware, only I loaded the aesni(4) driver instead of the ubsec(4) driver. This is is sort of an apples-to-oranges comparison, as cryptotest only tests AES with this driver, while the previous test was everything except AES.


```
[cmd=root@poolx:~ #]kldload crypto[/cmd]
[cmd=root@poolx:~ #]kldload cryptodev[/cmd]
[cmd=root@poolx:~ #]kldload aesni[/cmd]
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
[cmd=root@poolx:~ #]cryptotest -z 1024 16384[/cmd]
   0.063 sec,    2048    aes crypts,   16384 bytes, 532466350 byte/sec,  4062.4 Mb/sec
   0.072 sec,    2048 aes192 crypts,   16384 bytes, 468460664 byte/sec,  3574.1 Mb/sec
   0.081 sec,    2048 aes256 crypts,   16384 bytes, 416514796 byte/sec,  3177.8 Mb/sec
[cmd=root@poolx:~ #]cryptostats[/cmd]
67584 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations
[cmd=root@poolx:~ #]time cryptotest -z 102400 16384[/cmd]
   6.303 sec,  204800    aes crypts,   16384 bytes, 532364890 byte/sec,  4061.6 Mb/sec
   7.160 sec,  204800 aes192 crypts,   16384 bytes, 468654402 byte/sec,  3575.5 Mb/sec
   8.055 sec,  204800 aes256 crypts,   16384 bytes, 416565678 byte/sec,  3178.1 Mb/sec
0.067u 21.450s 0:21.51 100.0%   15+1522k 0+0io 0pf+0w
[cmd=root@poolx:~ #][/cmd]
```

This system happens to be a dual-processor system with 8 cores (E5620, hyperthreading turned off) which is a good bit faster than the average box that's out there.


----------



## do_more_secrets (May 14, 2013)

Terry_Kennedy said:
			
		

> Unfortunately, the CPU in the system you're using doesn't support aesni(4).



Unfortunately, your CPU and the CPU of my N40L are not equal in terms of speed and in terms of everything else.

At least in terms of speed, this is a real step forward


----------

