# About Nginx+HTTPS+SSL certs in Jails



## kalleboy (Jan 24, 2022)

Hi everyone. I'd like to set two jails with each hosting a domain of mine, with HTTPS/TLS support on nginx.

My case is;

My Dedicated Server/Host IP: 134.42.22.11 (External Public IP Addr) (has also PF activated and running without Jails' support, anything with any jail, at the moment)
Jail 1 - 10.10.10.2 - nginx: would host mydomain1.com - Port: 80/443
Jail 2 - 10.10.10.3 - nginx: would host myotherdomain.com - Port: 80/443

I'd like to know if not only HTTP but HTTPS traffic (and obviously SSL certs) is also supported. How to achieve this with nginx?

Would Nginx support HTTP/HTTPS redir, without using haproxy? Like using "proxy_pass" tags.

I generate my SSL certs by acme.sh, should I generate the SSL certificates within each jail or on the main host and put them into the jails' own related folders?

Best.


----------



## Hakaba (Jan 24, 2022)

The 'nginx' part is easy to set up. (I am on my phone, I will share a config if you need it)
I do not find a way to redirect traffic according to port+domain.
So a nginx as proxy (or HA Proxy) is needed. If there is a better solution, I am interested to.


----------



## kalleboy (Jan 24, 2022)

Hakaba said:


> The 'nginx' part is easy to set up. (I am on my phone, I will share a config if you need it)
> I do not find a way to redirect traffic according to port+domain.
> So a nginx as proxy (or HA Proxy) is needed. If there is a better solution, I am interested to.



Thanks a lot, well, I'm able to set nginx, with SSL support and virtualhosts and so on, on a real host. That's no problem.

I just want to learn the possibility of having them in each jail and, main host redirecting the HTTPS/TLS requests according to domain, to jails.


----------



## zirias@ (Jan 24, 2022)

So you want to use nginx as a reverse proxy for these two jails? The reverse proxy will then need both certificates (with private keys), but apart from that, a straight-forward config with two `server` blocks and the respective `server_name` properties will do, nginx supports SNI to determine the correct virtual server on TLS handshake.


----------



## kalleboy (Jan 24, 2022)

Well, imagine a jail with internal vLAN IP like: 192.168.1.2, and the nginx server conf is;


```
server {
    listen              192.168.1.2:443 ssl;
    server_name         www.example.com;
    ssl_certificate         www.example.com.crt;
    ssl_certificate_key www.example.com.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ...
}
```

How would such traffic pass to the external, public IP traffic of the host server?


----------



## zirias@ (Jan 24, 2022)

I think I answered that above? Typically with a reverse proxy listening on that public address, and you could also use nginx for that. And again, this reverse proxy will need all the certificates.

You might question whether you need TLS internally at all, depends on your scenario and threat models...

BTW, enabling anything older than TLS1.2 is probably not a good idea.


----------



## kalleboy (Jan 24, 2022)

Thanks Zirias. Oh, the sample code was from nginx documentation, I have only TLSv1.2 and TLSv1.3 activated, so no problem with that.

Hosted domains are not internal ones, so TLS requests would come from browsers of my website visitors.

So nginx would handle this itself with its proxy? Any example config in such case?

Thanks.


----------



## zirias@ (Jan 24, 2022)

kalleboy said:


> Hosted domains are not internal ones, so TLS requests would come from browsers of my website visitors.


That's not the issue. You want to host two different domains (with two different certificates) on a single IP address. This requires SNI, and therefore, the reverse proxy must terminate TLS anyways (and, needs all the certificates and private keys).

The proxied request will be in your local network (between your jails) only, so whether you need TLS there or not depends on your assessment of the threat of someone being able to eavesdrop connections inside your local network. On my private server, only my reverse proxy does TLS, requests are proxied using plain http internally.



kalleboy said:


> So nginx would handle this itself with its proxy? Any example in such case?


See above, it's straight forward. Add a `server` block for each domain on port 443 (and, if you want/need it, another one on port 80), inside add the correct `server_name`, the certificate configuration for the TLS servers, and some `proxy_pass` directive.


----------



## Hakaba (Jan 24, 2022)

there is more than on solution.
I have a jail that respond on 80 (for all acme domain) and 443 that resolve certificate and call non HTTPS internal jails ( port 8080, 8081 and so on) for page content depending on domain.
I change my mind and I create one jails by domain and on 'primary' jails to proxy pass the traffic.
(Between this two config, I use ovh-dns, but acme stop working without error...)
For me the second solution is easiest to handle and probably more robust. But if there is a tool that pass the trafic into a jail according to the domain name, this will be a better solution than the jail with nginx that made proxy pass...


----------



## SirDice (Jan 24, 2022)

Hakaba said:


> But if there is a tool that pass the trafic into a jail according to the domain name, this will be a better solution than the jail with nginx that made proxy pass...


net/haproxy (can do SSL termination too). On my VPS I have HAProxy running on the host and various websites in jails. HAProxy will proxy based on the URL to a specific jail backend.


----------



## jbo (Jan 24, 2022)

Maybe not answering any questions but I'd like to share how I handle this sort of things: I run a bunch of jails with www/nginx or other web servers in them (i.e. also www/gitea). None of these webservers does SSL. Instead, traffic is plain and I run net/haproxy in front of them which does SSL termination and redirects the plain text traffic to each jail based on the URL in the request header. This is easy to setup and easy to maintain.

*Edit:* Seems like SirDice covered this technique in the meantime - sorry.


----------



## angry_vincent (Jan 24, 2022)

can someone write step by step guide how to setup such jails plus nginx plus haproxy plus ssl? it would be much appreciated.


----------



## jbo (Jan 24, 2022)

angry_vincent said:


> can someone write step by step guide how to setup such jails plus nginx plus haproxy plus ssl? it would be much appreciated.


I'll get that done this week.


----------



## zirias@ (Jan 25, 2022)

It doesn't really matter WHICH reverse proxy you use, that's why I mentioned nginx will do just fine. Of course, haproxy is another alternative and offers additional features you might or might not need 

What's important is: In this scenario, hosting multiple TLS-domains on a single IP address, the reverse proxy MUST terminate TLS (and, therefore, must have all the certs and keys).


----------



## Hakaba (Jan 25, 2022)

Using HAProxy or NGinX for the proxy reverse is ok for me.
But I feel like a more elegant solution exists.
So my first approach was the good one, but as we lost HTTPS inside the jails, we have CORS errors in local browsing.


----------



## kalleboy (Jan 25, 2022)

Well, I guess when it comes to performance, it matters a lot;









						NGINX and HAProxy: Testing User Experience in the Cloud - NGINX
					

We compare the reverse proxying performance of HAProxy and NGINX. Performance is similar until the request rate is large enough for HAProxy to hit 100% CPU utilization. At that point, its performance degrades significantly while NGINX continues to experience almost no latency.




					www.nginx.com


----------



## zirias@ (Jan 25, 2022)

kalleboy operating at 100% CPU usage is not  a scenario you want in production anyways, so it doesn't matter too much. It's no surprise a scenario where nginx looks better is presented on nginx' site though 

Feature-wise, HAProxy is designed for "high availability", offering load balancing, down detection etc... if you need that, use HAProxy. Otherwise, it doesn't matter much, and IMHO, nginx is just fine.


----------



## jbo (Feb 2, 2022)

jbodenmann said:


> I'll get that done this week.


I'm like 50% done on this but got interrupted by real life stuff.
Will complete this next week.


----------



## kalleboy (Mar 8, 2022)

Sorry for the bump but any update in this jbodenmann ? Have you had any chance to complete it? 

Best wishes.


----------



## sko (Mar 9, 2022)

I usually run a single jail with nginx as reverse proxy for all webservers on that server (or network). This jails handles all SSL/TLS and cert stuff (or the jailhost is running acme.sh, depending on requirements), the other webservers only need a minimal non-SSL config.
This way I have only one "moving target" when it comes to certificates and all TLS-config is at one point, not scattered over probably dozens of jails/servers. This way it's easy to apply new TLS-configs and add/remove/modify domains/wildcards from certs.

As for the acme-http-validation: I abandoned this path a long time ago in favor of DNS-APIs. This way you can just 301 all requests on port 80 to port 443. Usually those "wellknown" configs on port 80 get very ugly very fast if you need to handle multiple domains and possibly even certs/acme clients on backend servers.
I use security/acme.sh everywhere; primarily because it doesn't drag in tons of dependencies like e.g. certbot, and because it is dead-simple to configure and automate, even for dozens of domains on the same host.


----------



## jbo (Mar 9, 2022)

kalleboy said:


> Sorry for the bump but any update in this _*[FONT=monospace]jbodenmann[/FONT]*_ ? Have you had any chance to complete it?


Yes, the blog post is pretty much complete. I'm still looking for someone to proof-read it. After that it just needs to be published (volunteers are welcomed).


----------



## Hakaba (Mar 10, 2022)

sko said:


> As for the acme-http-validation: I abandoned this path a long time ago in favor of DNS-APIs.


Unfortunately, this stop working one cold winter day with OVH DNS-API.
I do not find why and I configure the wellknow on port 80 in emergency.
I do not find the time to retest the DNS-API, so I let my server in this state.
Did you have a fallback ? Did you encounter issue with DNS-API ?


----------



## sko (Mar 10, 2022)

Hakaba said:


> Unfortunately, this stop working one cold winter day with OVH DNS-API.
> I do not find why and I configure the wellknow on port 80 in emergency.
> I do not find the time to retest the DNS-API, so I let my server in this state.
> Did you have a fallback ? Did you encounter issue with DNS-API ?


No, I don't use a fallback - the certs are renewed several days before they expire, so if it fails I can still react to the mail I receive from cron.

I only had a failure once when I accidentaly revoked the wrong api-key at digitalocean. Received an error message via email from cron 2 days in a row, so I intervened. The certs that were in place were still valid for several days, so no need to panic if it fails once or twice... (IIRC they are normally renewed 14 days prior to expiration).


----------



## kalleboy (Mar 30, 2022)

So, noone to test it, so far?


----------



## jbo (Apr 6, 2022)

kalleboy said:


> So, noone to test it, so far?


I'm not sure if that question is directed to me - in case it is: nope, I guess I'm just gonna release the blog post as-is and deal with complaints afterwards


----------



## wilschie (Apr 29, 2022)

jbodenmann said:
			
		

> I guess I'm just gonna release the blog post as-is and deal with complaints afterwards



Yes, please! 

Or I could test your documentation. Being new to technologies like Jails, HAProxy and/or VNET, even FreeBSD, something which brings all this together in one place would help a lot.


----------



## kalleboy (Jun 5, 2022)

Any progress on your documentation jbodenmann  ?


----------



## jbo (Jun 6, 2022)

kalleboy said:


> Any progress on your documentation jbodenmann  ?


Oh... good that you remind me of that.
I won't be able to publish it today but I'll get to it ASAP.


----------



## kalleboy (Jun 12, 2022)

jbodenmann said:


> Oh... good that you remind me of that.
> I won't be able to publish it today but I'll get to it ASAP.


No offense, but are you sure you have something written about it?  :/


----------



## getopt (Jun 12, 2022)

Jan 24, 2022


jbodenmann said:


> I'll get that done *this week*.


Feb 2, 2022


jbodenmann said:


> I'm like 50% done on this but got interrupted by real life stuff.
> Will complete this *next week*.


Mar 9, 2022


jbodenmann said:


> Yes, the blog post is *pretty much complete*. I'm still looking for someone to proof-read it. After that it just needs to be published (volunteers are welcomed).


Apr 6, 2022


jbodenmann said:


> I guess I'm just gonna *release the blog post as-is* and deal with complaints afterwards


Jun 6, 2022


jbodenmann said:


> Oh... good that you remind me of that.
> I won't be able to publish it today but I'll get to it* ASAP*.


Hmm. ... .... ..... ...... ....... ........ ?!

The P in ASAP stands for "possible".


----------



## jbo (Jun 12, 2022)

kalleboy said:


> No offense, but are you sure you have something written about it?  :/


Yes, it's currently being reviewed by a community member.
The post outlines the scenario, shows the relevant configuration of net/haproxy and provides the various shell scripts I created to manage the setup.



getopt said:


> The P in ASAP stands for "possible".


I too would have liked this to be different. As for any of us, there are responsibilities that need to be taken care of first. This is a personal blog post. I don't think that it gets much more lower priority than that. Surely nobody is actively waiting for this - everything I have to show is what you can get out of reading the various documentation:

I understand your frustration, I can assure you it is larger on my side.


----------



## getopt (Jun 12, 2022)

jbodenmann

You moved yourself in the position for being criticized publicly. That things developed this way is your responsibility of not doing what you promised.

You are justifying a timeline of disappointments. Furthermore you subtle mention a “forum member” as the possible point of failure, still reviewing but not finishing.



jbodenmann said:


> I too would have liked this to be different.


It’s not about you. Your audience liked it to be better.



jbodenmann said:


> As for any of us, there are responsibilities that need to be taken care of first.


First you promised “this week”, then “next week”. This builds responsibility to deliver. Not only within a professional context but in everyday situations. When speaking out such promises it is your responsibility to take into account your capabilities and willingness to complete as promised. You never communicated the level of your priority.



jbodenmann said:


> This is a personal blog post. I don't think that it gets much more lower priority than that.


Wow! That sentence is missing in your blog.



jbodenmann said:


> Surely nobody is actively waiting for this


How often does a person need to ask you, before you recognize that she is waiting/expecting something from you?



jbodenmann said:


> everything I have to show is what you can get out of reading the various documentation


Really? You cannot add anything to the docs? No hints? No tips? No experience? No professionalism? Hmm, ... then RTFM would be enough.

This already mounts to a beauty of failing but you make it even worse with a closing sentence like that:



jbodenmann said:


> I understand your frustration, I can assure you it is larger on my side.


Do you think that this is smart? It does not clean up anything, it mounts on top by posing that your frustration is even larger.

Cleaning up one’s mess in the public is certainly no business for amateurs. But hey, aren’t we here for learning?


----------



## Beastie7 (Jun 12, 2022)

Stop harassing the guy. You're entitled to nothing, and you're certainly aren't paying the person for _*volunteer*_ work either. It'll be done when he decides it's done. You want it expedited? Do the work yourself. Set proper expectations for yourself and you won't be so disappointed.


----------



## jbo (Jun 13, 2022)

FreeBSD: Simple Hosting
					

An introduction to hosting web services with FreeBSD.




					blog.insane.engineer


----------

