# FreeBSD bhyve VPS/Jail Hosting on single IP address.



## Norbert Szczybelski (Aug 31, 2020)

Good Evening. 

Are You Web Developer?

Working in Computer Service?

Got only one IP address?

You can start your own hosting server that will expand your offer.

Most of important things here are OpenVPN for Jail and bhyve SSH access and HAProxy for SNI extension based routing.

Your customer in bhyve/Jail can run any TCP/UDP service like Apache, Postfix, Dovecot, Asterisk or InspIRCd that are available through his OpenVPN Client.
Of course, ports can be redirected for direct connections.

Let’s Encrypt certificates are updating automatically by certbot software.

*bhyve HYPERVISOR (172.16.0.1) /etc/rc.conf:*

```
hostname="somehostname"
keymap="pl.kbd"
ifconfig_em0="172.16.0.1/12"
defaultrouter="172.31.255.254"
ntpdate_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"

sshd_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"

gateway_enable="YES"

vm_enable="YES"
vm_dir="zfs:data/bhyve"
vm_list="vm1 vm2jails vm3openvpn vm4haproxy vm5"

cloned_interfaces="bridge0 bridge1 bridge2 bridge3 bridge4 tap0 tap1 tap2 tap3 tap4"

ifconfig_bridge0="10.0.0.1/30 up"
ifconfig_bridge1="10.10.0.1/16 up"
ifconfig_bridge2="10.0.0.9/30 up"
ifconfig_bridge3="10.0.0.13/30 up"
ifconfig_bridge4="10.0.0.17/30 up"

autobridge_interfaces="bridge0 bridge1 bridge2 bridge3 bridge4"
autobridge_bridge0="tap0"
autobridge_bridge1="tap1"
autobridge_bridge2="tap2"
autobridge_bridge3="tap3"
autobridge_bridge4="tap4"

microcode_update_enable="YES"
```

*bhyve HYPERVISOR (172.16.0.1) /etc/pf.conf:*

```
external0 = "em0"
bridge0 = "bridge0:network"
bridge1 = "bridge1:network"
bridge2 = "bridge2:network"
bridge3 = "bridge3:network"
bridge4 = "bridge4:network"

nat on $external0 from $bridge0 to any -> ($external0)
nat on $external0 from $bridge1 to any -> ($external0)
nat on $external0 from $bridge2 to any -> ($external0)
nat on $external0 from $bridge3 to any -> ($external0)
nat on $external0 from $bridge4 to any -> ($external0)

rdr pass on $external0 inet proto tcp to port openvpn -> 10.0.0.10 port openvpn
rdr pass on $external0 inet proto tcp to port https -> 10.0.0.14 port https
rdr pass on $external0 inet proto tcp to port http -> 10.0.0.14 port http
rdr pass on $external0 inet proto tcp to port 50000 -> 10.0.0.18 port 50000

antispoof for em0
antispoof for bridge0
antispoof for bridge1
antispoof for bridge2
antispoof for bridge3
antispoof for bridge4

block in on $external0 all
pass out on $external0 all keep state

pass in on $external0 inet proto tcp from any to any port ssh
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/openvpn.conf*

```
mode server
tls-server
dev tun
proto tcp-server
port 1194

server 10.8.0.0 255.255.0.0

ca /root/certs/ca.crt
cert /root/certs/server.crt
key /root/certs/server.key
dh /root/certs/dh2048.pem
tls-crypt /root/certs/ta.key
crl-verify /crl.pem

cipher AES-256-CBC
auth SHA3-512

status /var/log/openvpn-status.log
log /var/log/openvpn.log
verb 5

user nobody
group nobody
persist-key
persist-tun
chroot /usr/local/etc/openvpn/jail
auth-nocache

duplicate-cn
client-to-client
client-config-dir /ccd

route 10.10.0.0 255.255.0.0
push "route 10.10.0.0 255.255.0.0"
```

*bhyve VM - vm3openvpn (10.0.0.10) /etc/pf.conf*

```
antispoof for vtnet0
antispoof for tun0

table <openvpn-server> persist {10.8.0.1}
table <openvpn-external-clients> persist {10.8.100.1,10.8.100.10,10.8.100.20}

block drop in on vtnet0 all
block drop in on tun0 all

pass in on tun0 inet proto tcp from <openvpn-external-clients> to <openvpn-server> port ssh

pass in on vtnet0 inet proto tcp from any to any port {ssh,openvpn}

pass out on vtnet0 all keep state
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/jail/ccd/jails*

```
iroute 10.10.0.0 255.255.0.0
ifconfig-push 10.8.0.10 10.8.0.11
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/jail/ccd/bhyve1*

```
ifconfig-push 10.8.0.20 10.8.0.21
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/jail/ccd/bhyve2*

```
ifconfig-push 10.8.0.30 10.8.0.31
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/jail/ccd/externalclient1*

```
ifconfig-push 10.8.100.1 10.8.100.2
```

*bhyve VM - vm3openvpn (10.0.0.10) /usr/local/etc/openvpn/jail/ccd/externalclient2*

```
ifconfig-push 10.8.100.10 10.8.100.11
```

*OPENVPN EXTERNAL CLIENT CONFIGURATION FILE*

```
client
remote-cert-tls server
dev tun

<connection>
remote vpn.domain.tld 1194 tcp-client
</connection>

cipher AES-256-CBC
auth SHA3-512

user nobody
group nogroup
persist-key
persist-tun
chroot /etc/openvpn/jail
auth-nocache

<ca>

</ca>
<cert>

</cert>
<key>

</key>
<tls-crypt>

</tls-crypt>
```

*bhyve VM - vm4haproxy (10.0.0.14) /usr/local/etc/haproxy.conf:*

```
global
daemon
maxconn 256

defaults
timeout client 30s
timeout server 30s
timeout connect 5s

frontend tlsproxy
bind 10.0.0.14:443
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend tls_proxy

backend tls_proxy
mode tcp

acl customname1 req_ssl_sni -i www.domain.tld
acl customname1redirect req_ssl_sni -i domain.tld
acl customname2 req_ssl_sni -i domain2.tld
acl customname3 req_ssl_sni -i domain3.tld
acl customname4 req_ssl_sni -i www.domain4.tld
acl customname4redirect req_ssl_sni -i domain4.tld

use-server server1 if customname1
use-server server1 if customname1redirect
use-server server2 if customname2
use-server server4 if customname3
use-server server5 if customname4
use-server server5 if customname4redirect

option ssl-hello-chk
server server1 10.0.0.6:443 check
server server2 10.0.0.2:443 check
server server4 10.10.0.5:443 check
server server5 10.10.0.6:443 check

frontend httpproxy
bind 10.0.0.14:80
mode http
default_backend http_proxy

backend http_proxy
mode http

acl customname1 hdr(host) -i www.domain.tld
acl customname1 hdr(host) -i domain.tld
acl customname2 hdr(host) -i domain2.tld
acl customname3 hdr(host) -i domain3.tld
acl customname4 hdr(host) -i www.domain4.tld
acl customname4 hdr(host) -i domain4.tld

use-server server1 if customname1
use-server server2 if customname2
use-server server3 if customname3
use-server server4 if customname4

server server1 10.0.0.6:80
server server2 10.0.0.2:80
server server3 10.10.0.5:80
server server4 10.10.0.6:80
```

*bhyve VM – vm4haproxy (10.0.0.14) /etc/pf.conf:*

```
antispoof for vtnet0
antispoof for tun0

block drop in on vtnet0 all
block drop in on tun0 all

pass out on vtnet0 all keep state
pass out on tun0 all keep state

pass in on vtnet0 inet proto tcp from any to any port {ssh,http,https}
pass in on tun0 inet proto tcp from 10.8.0.0/16 to any port ssh
```

*bhyve VM - vm2jails (10.10.0.2) /etc/pf.conf*

```
antispoof for vtnet0
antispoof for tun0

table <jails> persist {10.10.0.5,10.10.0.6,10.10.0.7}

block in on vtnet0 all
block in on tun0 all
block from 10.10.0.0/16 to 10.10.0.0/16

pass in on vtnet0 inet proto tcp from any to any port {ssh,http,https,openvpn}
pass in on tun0 inet proto tcp from 10.8.0.0/16 to <jails>

pass out on vtnet0 all keep state
pass out on tun0 all keep state
```

*bhyve VM - vm2jails (10.10.0.2) CRON JOB*

```
*/5 * * * * service pf reload
```

*bhyve VM - vm2jails (10.10.0.2) /etc/jail.conf*

```
jail1 {
    host.hostname = "domain3.tld";
    interface = "vtnet0";
    ip4.addr = "10.10.0.5";
    path = "/jails/customer1/";
    mount.devfs;
    allow.raw_sockets;
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
}

jail2 {
    host.hostname = "www.domain4.tld";
    interface = "vtnet0";
    ip4.addr = "10.10.0.6";
    path = "/jails/customer2/";
    mount.devfs;
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
}

jail3 {
    host.hostname = "otherhostname";
    interface = "vtnet0";
    ip4.addr = "10.10.0.7";
    path = "/jails/customer3/";
    mount.devfs;
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
}
```

*SOME CUSTOMER JAIL (10.10.0.6) IN vm2jails /usr/local/etc/haproxy.conf*

```
global
daemon
maxconn 256

defaults
timeout client 30s
timeout server 30s
timeout connect 5s

frontend http_to_tls_redirection
bind *:80
mode http

acl customname hdr(host) -i www.domain4.tld
acl customname hdr(host) -i domain4.tld
http-request redirect location https://www.domain4.tld code 301 if customname
```

*SOME CUSTOMER JAIL (10.10.0.6) IN vm2jails /usr/local/etc/apache24/sites/domain4.tld.443.conf*

```
<VirtualHost *:443>
ServerName domain4.tld
DocumentRoot "/usr/local/www/apache24/data/www.domain4.tld/wordpress/"

Redirect 301 / https://www.domain4.tld

SSLEngine on

SSLCertificateFile "/usr/local/etc/letsencrypt/live/www.domain4.tld/fullchain.pem"
SSLCertificateKeyFile "/usr/local/etc/letsencrypt/live/www.domain4.tld/privkey.pem"

</VirtualHost>
```

*SOME CUSTOMER JAIL (10.10.0.6) IN vm2jails /usr/local/etc/apache24/sites/www.domain4.tld.443.conf*

```
<VirtualHost *:443>
ServerName www.domain4.tld
DocumentRoot "/usr/local/www/apache24/data/www.domain4.tld/wordpress/"

SSLEngine on

SSLCertificateFile "/usr/local/etc/letsencrypt/live/www.domain4.tld/fullchain.pem"
SSLCertificateKeyFile "/usr/local/etc/letsencrypt/live/www.domain4.tld/privkey.pem"

<Directory /usr/local/www/apache24/data/www.domain4.tld/wordpress/>
Options -Indexes +FollowSymLinks +MultiViews
AllowOverride All
Require all granted
</Directory>

</VirtualHost>
```

*SOME CUSTOMER JAIL (10.10.0.6) IN vm2jails CRON JOB*

```
0 4 * * * certbot renew --pre-hook "service apache24 stop && service haproxy stop" --post-hook "service apache24 start && service haproxy start"
```

Based on FreeBSD 12.2 RELEASE AMD64 systems.

Any improvements are welcome.

Like for me works great.

Thanks for FreeBSD. 

Norbert.


----------



## zennybsd (Jan 26, 2021)

Thanks Norbert,

Btw, it would be nice if you share your network topology if any.

Meanwhile, have you tried to deploy microservices like https://github.com/msimerson/Mail-Toaster-6 for provisioning?


----------



## Norbert Szczybelski (Feb 6, 2021)

Good morning zennybsd. 

Attaching scheme in server-room.png
Should speed up imagination.

I have plans add SMTP and IMAP servers to this tutorial.
But I must test them on my infrastructure first.

Norbert.


----------

