# Nginx KTLS support on FreeBSD 13/13.1?



## kalleboy (Jan 6, 2022)

Today did a pkg upgrade and my nginx updated to: nginx-devel-1.21.5_3 on FreeBSD 13, bringing support to KTLS.

Tried to activate the KTLS within the kernel with command:
`kldload ktls_ocf`
then: `sysctl kern.ipc.tls.enable=1`

afterwards, I added; `ssl_conf_command Options KTLS;`" under SSL server block of my nginx.conf file, tried to refresh nginx, got an error like;

```
nginx: [emerg] SSL_CONF_cmd("Options", "KTLS") failed (SSL: error:1414E180:SSL routines:SSL_CONF_cmd:bad value:cmd=Options, value=KTLS)
```


```
root@:~ # uname -a
FreeBSD myhost 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 [EMAIL]root@amd64-builder.daemonology.net[/EMAIL]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
```

Could anyone point me please where do I make mistake?

Best.


----------



## covacat (Jan 6, 2022)

nginx docs say you have to build nginx with openssl 3.0


----------



## kalleboy (Jan 6, 2022)

Hmm, then I'll have to stop using nginx through packages? That was a bad news


----------



## kalleboy (Jan 6, 2022)

covacat Nginx docs say: 
"FreeBSD 13.0+. As of November 2021, FreeBSD 13.0+ is the only OS that supports kTLS in NGINX *without a manual build of NGINX to incorporate OpenSSL 3.0.0+*"


----------



## covacat (Jan 6, 2022)

options KTLS seems to be passed to the ssl lib and it does not know about it
i look the Base openssl and it's not there
openssl 1.1m from ports has it


----------



## kalleboy (Jan 7, 2022)

covacat said:


> openssl 1.1m from ports has it



Could that be installed as a pkg, or any such solution? Haven't messed with any ports on my system yet.


----------



## covacat (Jan 7, 2022)

i don't think it is available as a package
can you verify ktls is working if you remove nginx options ktls ?
it may be activated by default ?


----------



## sko (Jan 7, 2022)

kalleboy said:


> covacat Nginx docs say:
> "FreeBSD 13.0+. As of November 2021, FreeBSD 13.0+ is the only OS that supports kTLS in NGINX *without a manual build of NGINX to incorporate OpenSSL 3.0.0+*"



from the history of www/nginx:


> www/nginx: Update to 1.20.2
> 
> Changes:
> *) Feature: OpenSSL 3.0 compatibility.



I suspect the note in the manpage only accounts to the nginx-side of OpenSSL 3.0 / KTLS support - i.e. "nginx is built with OpenSSL 3.0 support enabled on FreeBSD 13, but not on other OS yet". It does not explicitly say that OpenSSL 3.0 and/or KTLS is already supported on FreeBSD in base.
Maybe you could try installing security/openssl-devel, which is currently version 3.0.1. But I wouldn't feel good about running a development version of something as crucial as OpenSSL...

Are you hitting any performance limits or problems with the way TLS is currently handled (i.e. not in kernel)? If not I'd wait until this is battle-tested and becomes supported in base or at least with an OpenSSL release-version from pkg/ports, not a development version (at least for anything remotely related to a production system and/or connected to the outside world).


edit:
according to this commit https://reviews.freebsd.org/D24274 it should already be available in base with OpenSSL 1.1.1


----------



## cmoerz (Jan 7, 2022)

You can enable KTLS in base OpenSSL by setting

```
WITH_OPENSSL_KTLS=1
```
in /etc/src.conf. KTLS is otherwise already available (but not enabled for connections) by default in the GENERIC kernel, if I'm not mistaken.

I ran into related issues recently when attempting to set up NFS over TLS (see https://forums.freebsd.org/threads/kerberized-nfsv4-nfs-over-tls-on-13-0.83484/#post-548399)


----------



## kalleboy (Jan 7, 2022)

So it seems creating a src.conf file with WITH_OPENSSL_KTLS=1 line, then moving from 13-RELEASE to 13-STABLE (compiling from sources) the best option to enable KTLS in Nginx on FreeBSD 13?

and to answer sko , indeed, I'd like to have it due to performance purposes.

Best.


----------



## cmoerz (Jan 8, 2022)

From what I gather, WITH_OPENSSL_KTLS=1 works on RELEASE as well; it requires the applications to be tailored for it, however. So - without having it tried myself - I suppose, chances are good that nginx's KTLS works the same principle as the port I tried - and that did require STABLE to work properly.

Rick Macklem's documentation explicitly pointed out that running with WITH_OPENSSL_KTLS will likely break applications like apache; with nginx "understanding" KTLS, I expect chances are good with RELEASE and still better with STABLE since their handling of connections appear to be different.


----------



## Bink (May 23, 2022)

Just for those looking into this a bit later (as I am), it appears WITH_OPENSSL_KTLS is now the default in FreeBSD 13.1 RELEASE for amd64/amd64 and arm64/aarch64.

Source:





						src.conf
					






					www.freebsd.org


----------



## Bink (May 26, 2022)

See notes here too, about how to enable KTLS in Nginx, and enable Kernel TLS on FreeBSD 13:
https://cgit.freebsd.org/ports/commit/?id=62502a50558bbc0950693a9ea52aee738a80f0c8



> New kernel TLS feature is available starting with FreeBSD 13.0, and it requires OpenSSL 3.0, compiled with "enable-ktls" option.
> Further, KTLS needs to be enabled in kernel, and in OpenSSL, either via OpenSSL configuration file or with
> 
> 
> ...


----------



## drhowarddrfine (May 26, 2022)

Fun fact: nginx was originally developed exclusively on FreeBSD.


----------



## kalleboy (Jul 28, 2022)

Bink thanks for the link.

Under FreeBSD 13.1, I don't get any more the error at the top of my post, from nginx. The ssl_conf_command Options KTLS; works now. But;

It seems KTLS needs `sendfile on;` for nginx. BUT sendfile is not recommended/supported when you're on *ZFS*.

Please see: https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/

And I think KTLS isn't get enabled when sendfile is off, even though you did anything else for it;

```
kern.ipc.tls.enable=1
kern.ipc.mb_use_ext_pgs=1
kldload ktls_ocf
kld_list="ktls_ocf"
```

Using both commands return nothing from Nginx (in debug mode);
`grep BIO /var/log/nginx/error.log`
`grep SSL_sendfile /var/log/nginx/error.log`


So, how can I proceed with KTLS while sendfile is off in nginx, under ZFS?


----------



## kalleboy (Jul 28, 2022)

Or I better should ask, does the following nginx package (on FreeBSD 13.1) supports KTLS with the mentioned OpenSSL version?

nginx version: nginx/1.22.0
built with OpenSSL 1.1.1k-freebsd 24 Aug 2021 (running with OpenSSL 1.1.1o-freebsd 3 May 2022)


----------



## suntzu00 (Jul 28, 2022)

ktls(4)
					






					www.freebsd.org
				




OpenSSL 3.0 and later include support for *ktls*.  The _security/openssl-devel_ port may also be built with support for *ktls* by enabling the KTLS option. OpenSSL in the base system includes KTLS support when built with WITH_OPENSSL_KTL.


----------



## kalleboy (Jul 28, 2022)

I didn't really get it suntzu00

Obviously, FreeBSD 13.1's default base OpenSSL is not 3.0. BUT the link says the base one is compiled with KTLS support?


----------



## kalleboy (Jul 29, 2022)

By the way, here's my sysctl -a | fgrep kern.ipc.tls.stats output;

https://bsd.to/Hrgx
Does this indicate that KTLS works?


----------

