# FreeBSD VPS Jailed Web Servers Network Isolation



## scrappywan (Jul 20, 2016)

I have a VPS on Digitalocean which I used mfsbsd to reinstall FreeBSD with ZFS/zroot with PF as my firewall. My plan with this VPS is to run wordpress, a static site and owncloud each in their own jails. Currently, I use nginx on the host machine running as a reverse proxy, intercepting https traffic and directing the unencrypted final bits to the appropriate jails. While this works and the outside world can connect to https://site-example-1.com, https://site-example-2.com etc., because the traffic going from host to jail isn't encrypted I worry that a compromised jail could intercept the unencrypted traffic of another jail. My jails are currently networked on a lo1 cloned interface with aliases and when I checked, each jail is able to access the other's running nginx service. Even after changing jails to different subnets (172.16.1.1 -jail A, 172.16.2.1 -jail B), jail A is able to see jail B's web server. I assume this is because they are all on the same loopback interface. 

So, my question is: is there a best practice for isolating jails from being able to talk to one another? Can this be done through NAT via pf rules? Or, is there a better way to forbid jails from knowing another jail exists on the host machine? Or or, is running jails as fully contained web servers a dumb idea altogether?

While I am an enthusiast of FreeBSD and other *nix systems, I am one of those people who know just enough to sometimes break things. I don't have the best understanding of networking or firewalls so any help in this area would be greatly appreciated.


----------



## Remington (Jul 20, 2016)

Try this and see if it helps.  As far as I know only local network such as 'lo' works.  Using interface card will not work.

https://forums.freebsd.org/threads/17265/


----------



## scrappywan (Jul 21, 2016)

Remington said:


> Try this and see if it helps.  As far as I know only local network such as 'lo' works.  Using interface card will not work.
> 
> https://forums.freebsd.org/threads/17265/



Thanks for the link. It is helping me understand things better but I have yet to figure it out. What the user in your link found to work is not working for me for some reason.

This is my current rc.conf:

```
hostname="zim"
ifconfig_vtnet0="inet 5.6.7.8 netmask 0xffffc000"
defaultrouter="5.6.128.1"
sshd_enable="YES"
powerd_enable="YES"

ntpd_enable="YES"
ntpdate_enable="YES"
ntpd_sync_on_start="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
clear_tmp_enable="YES"

iocage_enable="YES"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

gateway_enable="YES"

# Setup interface jails will use:
cloned_interfaces="lo1"
ifconfig_lo1="inet 172.16.1.1 netmask 255.255.255.0"
ifconfig_lo1_alias0="inet 172.16.1.2 netmask 255.255.255.255"
ifconfig_lo1_alias1="inet 172.16.1.3 netmask 255.255.255.255"
ifconfig_lo1_alias2="inet 172.16.1.4 netmask 255.255.255.255"

sshguard_enable="YES"
sshguard_watch_logs="/var/log/auth.log"
sshguard_safety_thresh="30"
sshguard_pardon_min_interval="1200"
sshguard_prescribe_interval="7200"

nginx_enable="YES"
```

My current pf.conf with several lines commented out that didn't work to block http traffic between jails:


```
# Name the interfaces
ext_if = "vtnet0"
int_if = "lo0"
jail_if = "lo1"
ip_pub = "5.6.7.8" # VPS Static IP
tcp_services = "{ 22 53 80 443 }"
udp_services = "{ 53 }"
jail_net = "172.16.1.0/24"
port_jail = "{ 80 443 }"
kloudj_net = "172.16.1.3" #jail1
tekj_net = "172.16.1.2" #jail2
testj_net = "172.16.1.4" #jail3
zefj_net = "172.16.1.1" #jail4

#set skip on lo

scrub in all

#Define the NAT for the jails
nat on $ext_if from $jail_net to any -> $ip_pub

# Let sshguard tell pf what to block
table <sshguard> persist
block in quick on vtnet0 proto tcp from <sshguard> to any port 22 label "ssh bruteforce"

# Default secure
block in all

# Let sshguard tell pf what to block
table <sshguard> persist
block in quick on vtnet0 proto tcp from <sshguard> to any port 22 label "ssh bruteforce"

# Default secure
block in all
pass out all keep state

### Prevent Jails from seeing each other's http traffic
#pass in quick on $jail_if inet proto tcp from ! $jail_if to $jail_if port http ## Did not isolate jails
#pass quick on lo0 inet proto tcp from $jail_net to $jail_net port http keep state ## Did not isolate jails
#block quick on lo0 inet proto tcp from $jail_net to $jail_net port http ## Did not isolate jails
#pass quick on lo0 inet proto tcp from $testj_net to $tekj_net port 80 ## Did not isolate those jails

# Allow services listed in tcp_services and udp_services in through firewall
pass in proto tcp to any port $tcp_services keep state
pass in proto udp to any port $udp_services keep state
```

I have tried the same lines with pass and block. Also tried using lo, lo0, lo1 but nothing works.

Whatever the issue is, I would venture to guess it's fairly simple and staring me right in the face. But for now I remain puzzled :/


----------



## SirDice (Jul 21, 2016)

The "problem" is that packets need to traverse an interface for PF to be able to do something. Packets traveling from one jail to the other never traverses the interface, it's all local, so PF is never able to filter it.

You might be able to filter "inter-jail" traffic if you use VNET/VIMAGE jails. Don't know for sure though, never tested it myself.


----------



## kpa (Jul 21, 2016)

You can use multiple interfaces, bind each jail to a different interface and then write firewall rules to limit traffic between the interfaces and the host or the internet. Cloned lo(4) interfaces should work, if not you can definitely do the isolation using multiple tap(4) interfaces.


----------



## scrappywan (Jul 21, 2016)

SirDice said:


> The "problem" is that packets need to traverse an interface for PF to be able to do something. Packets traveling from one jail to the other never traverses the interface, it's all local, so PF is never able to filter it.
> 
> You might be able to filter "inter-jail" traffic if you use VNET/VIMAGE jails. Don't know for sure though, never tested it myself.



What you say about PF not being able to filter in the same interface makes sense. I even considered this earlier as to why nothing works to block traffic between jails with my current setup. Still trying to learn PF and this makes for a good learning experience.

I have been hesitant to try VNET/VIMAGE due to the understanding it is not very stable yet. However, in iocage's documentation it says if you follow certain guidelines with VIMAGE then you might be Ok with it.


----------



## scrappywan (Jul 21, 2016)

If I did away with the nginx reverse proxy on my host machine, moved my SSL certs from the host machine to the appropriate jails and used PF to redirect all incoming http/https traffic from host to the jail NAT couldn't each jail running its own web server listen to, and receive the correct traffic based upon http host headers?


----------



## Remington (Jul 21, 2016)

Okay... I played around with pf and got it working by blocking all traffic on 'lo' and opened one way traffic to one jail.  You can replace { ssh } with { ssh http https }.  I stripped out much of the unrelated lines of code to keep this short and easy to follow.

Here is my pf.conf


```
# Interfaces
ext_if  = "igb0"
ext_addr  = "192.168.1.100"  # Host address

int_if  = "lo1"
int_addr  = "10.0.0.0/24"  # Local addresses

# Normalization: reassemble fragments etc
scrub in all

# NAT
nat on $ext_if from $int_addr to any -> ($ext_if)

# NAT rules
###########################################################

# Host
rdr on $ext_if proto { tcp } from any to $ext_if port { ssh }  -> $ext_addr  # ssh

# Filtering rules
###########################################################

# Block in
block in all

# Pass in/out - Host - SSH
pass quick on $ext_if proto { tcp } from any to $ext_addr port { ssh } keep state

# Pass ini/out - Jail - Allow SSH connection from 10.0.0.2 to 10.0.0.1
pass quick on $int_if proto { tcp } from 10.0.0.2 to 10.0.0.1 port { ssh }
```


----------



## SirDice (Jul 21, 2016)

> If I did away with the nginx reverse proxy on my host machine, moved my SSL certs from the host machine to the appropriate jails and used PF to redirect all incoming http/https traffic from host to the jail NAT couldn't each jail running its own web server listen to, and receive the correct traffic based upon http host headers?


Not possible as PF only works on layer 3/4, so it has no idea about the "Host:" header. Instead of nginx I can highly recommend net/haproxy. But you'd still have the issue of jails being able to access the other jails.


----------



## SirDice (Jul 21, 2016)

Remington said:


> Okay... I played around with pf and got it working by blocking all traffic on 'lo' and opened one way traffic to one jail.  You can replace { ssh } with { ssh http https }


I might be misreading it but your pf.conf isn't blocking anything between two jails.


----------



## Remington (Jul 21, 2016)

SirDice said:


> I might be misreading it but your pf.conf isn't blocking anything between two jails.



When I comment this out then SSH won't be able to connect so jail to jail communication is effectively blocked.


```
# Pass ini/out - Jail - Allow SSH connection from 10.0.0.2 to 10.0.0.1
pass quick on $int_if proto { tcp } from 10.0.0.2 to 10.0.0.1 port { ssh }
```

I didn't include the following codes which are commonly used:

```
set skip on lo
pass out all keep state
pass quick on lo
pass in quick on lo all
```

I understand that `set skip on lo` skips filtering rules on loopback interface.  So commenting it out effectively forces the rules on loopback interfaces.


----------



## Remington (Jul 21, 2016)

If you want two-way communications between jails then you will set something like this.


```
# Pass ini/out - Jail- Allow SSH between two jails.
pass quick on $int_if proto { tcp } from 10.0.0.1 to 10.0.0.2 port { ssh }
pass quick on $int_if proto { tcp } from 10.0.0.2 to 10.0.0.1 port { ssh }
```


----------



## scrappywan (Jul 21, 2016)

After a bunch of trial and error with the advice from Remington, this is my current pf.conf:

```
# Name the interfaces
ext_if="vtnet0"
int_if="lo0"
jail_if="lo1"
ip_pub="5.6.7.8"
tcp_services="{ 22 53 80 443 }"
udp_services="{ 53 }"
jail_net="172.16.1.0/24"
port_jail="{ 80 443 }"
kloudj_net="172.16.1.3"
tekj_net="172.16.1.2"
testj_net="172.16.1.4"
zefj_net="172.16.1.1"

scrub in all

#Define the NAT for the jails
nat on $ext_if from $jail_net to any -> ($ext_if)

rdr on $ext_if proto { tcp } from any to $ext_if port { http }  -> $ip_pub  # http

# Let sshguard tell pf what to block
table <sshguard> persist
block in quick on vtnet0 proto tcp from <sshguard> to any port 22 label "ssh bruteforce"

# Default secure
block in all
#pass out all keep state

# Prevent Jails from seeing each other's http traffic
pass in quick on $jail_if inet proto tcp from $jail_if to $jail_if port { http }

# Allow services listed in tcp_services and udp_services in through firewall
pass in proto tcp to any port $tcp_services keep state
pass in proto udp to any port $udp_services keep state
```

Performing a dry run of PF rules I get this:

```
root@zim:/etc # pfctl -vnf /etc/pf.conf                                                                       
ext_if= "vtnet0"
int_if = "lo0"
jail_if = "lo1"
ip_pub = "5.6.7.8"
tcp_services = "{ 22 53 80 443 }"
udp_services = "{ 53 }"
jail_net = "172.16.1.0/24"
port_jail = "{ 80 443 }"
kloudj_net = "172.16.1.3"
tekj_net = "172.16.1.2"
testj_net = "172.16.1.4"
zefj_net = "172.16.1.1"
table <sshguard> persist
scrub in all fragment reassemble
nat on vtnet0 inet from 172.16.1.0/24 to any -> (vtnet0) round-robin
rdr on vtnet0 inet proto tcp from any to 5.6.7.8 port = http -> 5.6.7.8
block drop in quick on vtnet0 proto tcp from <sshguard> to any port = ssh label "ssh bruteforce"
block drop in all
pass in quick on lo1 inet proto tcp from 172.16.1.3 to 172.16.1.3 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.3 to 172.16.1.1 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.3 to 172.16.1.2 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.3 to 172.16.1.4 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.1 to 172.16.1.3 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.1 to 172.16.1.1 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.1 to 172.16.1.2 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.1 to 172.16.1.4 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.2 to 172.16.1.3 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.2 to 172.16.1.1 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.2 to 172.16.1.2 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.2 to 172.16.1.4 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.4 to 172.16.1.3 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.4 to 172.16.1.1 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.4 to 172.16.1.2 port = http flags S/SA keep state
pass in quick on lo1 inet proto tcp from 172.16.1.4 to 172.16.1.4 port = http flags S/SA keep state
pass in proto tcp from any to any port = ssh flags S/SA keep state
pass in proto tcp from any to any port = domain flags S/SA keep state
pass in proto tcp from any to any port = http flags S/SA keep state
pass in proto tcp from any to any port = https flags S/SA keep state
pass in proto udp from any to any port = domain keep state
```

This successfully blocks traffic between jails but unfortunately broke http traffic from my host machine out.


----------



## Remington (Jul 21, 2016)

I noticed that when I tried to SSH remotely to one of the jail and it won't connect.  I'm working on different pf configurations.


----------



## scrappywan (Jul 21, 2016)

I should say that I can access external websites from within my jails, but the host vps has no ability to access any http traffic probably because of
`#pass out all keep state` being commented out which breaks my jail isolation as soon as I reinstate it in my pf rules.


----------



## Remington (Jul 21, 2016)

scrappywan said:


> I should say that I can access external websites from within my jails, but the host vps has no ability to access any http traffic probably because of
> `#pass out all keep state` being commented out which breaks my jail isolation as soon as I reinstate it in my pf rules.



I think everything else should be okay except that one `set skip on lo` should be commented out if you want to keep jails from talking to each other.


----------



## scrappywan (Jul 21, 2016)

Remington said:


> I think everything else should be okay except that one `set skip on lo` should be commented out if you want to keep jails from talking to each other.


Thing is, I do not have the line `set skip on lo` in my pf.conf and my host is still unable to connect to the outside world --only the jails can with the above config.


----------



## SirDice (Jul 22, 2016)

scrappywan said:


> I should say that I can access external websites from within my jails, but the host vps has no ability to access any http traffic probably because of
> `#pass out all keep state` being commented out which breaks my jail isolation as soon as I reinstate it in my pf rules.


You could try something like this:

```
pass out from any to ! 10.0.0.0/8 keep state
```


----------



## scrappywan (Jul 22, 2016)

SirDice said:


> You could try something like this:
> 
> ```
> pass out from any to ! 10.0.0.0/8 keep state
> ```



Gave it a shot `pass out from any to ! $jail_net keep state`. While it gives me host http access out, this ends up blocking me from accessing my jailed webserver from outside the jail (and from within the jail for that matter). It did, however, work to isolate jails from seeing one another.

Perhaps what I have been trying to accomplish this whole time is simply not meant to be. Maybe as mentioned earlier, maybe I should look into bridge/tap instead?


----------



## SirDice (Jul 22, 2016)

Lets go back to what you're trying to accomplish. Is it just that one jail should not be able to access the other jail's website? If that's the case have you considered simply blocking access on the webserver itself? A fairly basic ACL on nginx can be used to block traffic from any of the other jails while allowing access from everywhere else. 

But I'm wondering if you're not trying to over think things. If both sites are publicly accessible what harm could there be if one jail accesses the website of another jail directly?


----------



## scrappywan (Jul 22, 2016)

SirDice said:


> Lets go back to what you're trying to accomplish. Is it just that one jail should not be able to access the other jail's website? If that's the case have you considered simply blocking access on the webserver itself? A fairly basic ACL on nginx can be used to block traffic from any of the other jails while allowing access from everywhere else.
> 
> But I'm wondering if you're not trying to over think things. If both sites are publicly accessible what harm could there be if one jail accesses the website of another jail directly?



My primary concern is that the host vps stores my ssl certs for the multiple websites I plan to host (one fully contained site per jail). As it is currently set up, the host decrypts the ssl/https traffic and passes it along via nginx reverse proxy --unencrypted, to whichever jail is hosting that particular site. I don't want the jails to see each other at least on port 80 in case one is ever compromised.

Perhaps I am going about this totally wrong. if you, or anybody else could suggest a best practice to keep sites isolated through jails I am all ears.

I do appreciate all the help so far. At the very least this has been a good learning experience.


----------



## priyadarshan (Oct 15, 2016)

Talking as a jails and pf absolute beginner, I would probably try moving `nginx` to inside each jail. It is a little more maintenance work, but it would assure complete isolation of unencrypted traffic.

If I would have very stringent security requests, depending on the maintenance budget of course, I would perhaps even use `bhyve`.


----------



## storvi_net (Oct 15, 2016)

I assume because of the missing multiple public IPs this will not work.
The SSL-Termination must take place as near as possible to the user, because only reverse proxy has access to the "HOST"-Field.


----------

