# IPFW and Virtualized Jail



## beaute (Aug 3, 2010)

Hello, 

I have a FreeBSD box with two jails in it. Each jail has two interfaces. I also have a bridge residing on the main machine to connect one of the two interfaces of the jails together. 

Jail 1 has interface A: 192.168.1.2 and Interface B: 10.0.0.1
Jail 2 has interface C: 192.168.1.3 and Interface D: 172.16.0.1

Host Machine has many interfaces but the most important ones right now are
bridge0: 192.168.1.1 so that interface A and Interface C can ping each other. 
msk0: 1.1.1.1
em0: 2.2.2.2

I wish to strengthen my scheme using firewall. However, when I execute the firewall settings I seem to get a block on traffic between the jails with Permission Denied and I can't seem to figure out why. I get this error when trying to ping the host from interface A. When I ping interface A from the host, it just halts like it's expecting a return which is being dropped. So I'm assuming echo replies are not passing through. 

Other than this, I would eventually like to bind interface B to em0, so that those two are forwarding traffic to each other and only to each other. I really don't know how to do this. When I'm not using a firewall, everything can reach everything as it is supposed to be. I just want to make sure everything is only reaching what it's supposed to be reaching. 

I'm sure the solution is simple, I'm just new to ipfw (and FreeBSD).

Here are my rules so far:

```
#!/bin/sh
ipfw -q -f flush

# Set Defaults
ks="keep-state"
cmd="ipfw -q add"
pif="msk0"

# No restriction on loopback
$cmd 005 allow all from any to any via lo0

# Allow packet through if previously added
$cmd 010 check-state

# Allow incoming and outgoing ping to the main interface
$cmd 015 allow icmp from any to any out via $pif $ks
$cmd 016 allow icmp from any to any in via $pif $ks

# Allow IPSec
$cmd 017 allow log esp from any to any
$cmd 018 allow log ah from any to any
$cmd 019 allow log ipencap from any to any
$cmd 020 allow log udp from any 500 to any

# Allow connection to/from the jail
$cmd 021 allow icmp from 192.168.1.0/24 to any in
$cmd 022 allow icmp from any to 192.168.1.0/24 in

#Deny Everything else by default
$cmd 999 deny log all from any to any
```


Oh and finally, do I need NAT? I'm not really clear what NATing is (yea network address translation but i don't really know what it does) so i just assumed that I don't need it. all my IP addresses are fixed and I'm working in a controlled LAN and don't need internet access of any sort. The reason I ask is that maybe all of this failing because I have not compiled my kernel with IPDIVERT. So I'm wondering...


Thanks a lot in advance


----------



## fbsd1 (Aug 5, 2010)

Looks like ip address assignment problem. 

Interface a and c have to be on their own NIC device. 
And be assigned on different octo number. 

interface NIC A: 192.168.1.2 
interface NIC C: 192.168.2.3 

And jail should really be auto creating a alias on 
start  and auto removing alias on stoping. 

I dont think your problem is the host firewall,
but more a matter of how you have your jails configured.

Jails can not have firewalls.


----------



## beaute (Aug 5, 2010)

fbsd1 said:
			
		

> Jails can not have firewalls.


I am really surprised that you say this. Because adding ipfw rules to the jail have actually allowed me to solve the problem. So I'm very surprised. 

I am now able to ping between one of the jail's interfaces to the other jail's interface and to the bridge. Basically, the following pings work:
`# ping 192.168.1.2`
`# ping 192.168.1.3`
`# jexec 1 ping 192.168.1.1`
`# jexec 1 ping 192.168.1.3`

What I would like to do now is make a rule to force traffic going out from the jail to the2.0.0.0/8 network to use the jail interface with the IP address 10.0.0.1 and only that. Anything coming from 10.0.0.1 will only be sent to hosts in the 2.0.0.0/8 subnet. 

It would be the same for the other interface in the other jail (172.16.0.1 would only send to the 3.0.0.0/8 subnet). How can I do this? I have enabled IPFW Forward and IPFW Divert in my kernel but I don't even know if I have to use them or no. 

Here is a simplified ASCII of my scheme. 

```
IP Cipher 1	
	 _______________________
	|			|
	|    ______________	|
	|   |		   |	|
	|   |	  10.0.0.1>x<-->em0 2.2.2.2 <--------------------------------> 2.2.2.3
	|   |	  ipfw	 | |	|
	|   |  192.168.1.2>i<	|
	|   |______________| \	|
	|		      |	|
	|		      |	|
	|		      |	|
	|	 	      |	|
	|		      \ |	
	|	 ( ipfw )      >bridge0 192.168.1.1
	|		      / |
	|		      |	|
	|		      |	|
	|		      |	|
	|    ______________   |	|
	|   |		   | /	|
	|   | 192.168.1.3>j<	|
	|   |	  ipfw	 | |	|
	|   |	172.16.0.1>y<-->em1 3.3.3.2 <--------------------------------> 3.3.3.3
	|   |______________|	|
	|			|
	|_______________________|
```


My ipfw.rules on the host machine

```
####################################
###########    HOST A	############
####################################

#!/bin/sh
# Start of IPFW Rules File
ipfw -q -f flush		# Flush rules

# Set defaults
ks="keep-state"
cmd="ipfw -q add"		# Add new rule
pif="msk0"			# Public interface name
bridge="bridge0"		# Bridge interface
int_if1="em0"			# First private interface

# No restriction on loopback
$cmd 0005 allow all from any to any via lo0

# Allow packet through if previously added
$cmd 0010 check-state

# Allow incoming and outgoing ping to the main interface
$cmd 015 allow icmp from any to any out via $pif $ks
$cmd 016 allow icmp from any to any in via $pif $ks

# Allow IPSec
$cmd 017 allow log esp from any to any
$cmd 018 allow log ah from any to any
$cmd 019 allow log ipencap from any to any
$cmd 020 allow log udp from any 500 to any

# Allow incoming and outgoing ICMP on the bridge
$cmd 0030 allow icmp from any to any out via $bridge $ks
$cmd 0031 allow icmp from any to any in via $bridge $ks

# Allow ICMP from friendly subnet
$cmd 0040 allow icmp from 2.0.0.0/8 to me
$cmd 0041 allow icmp from me to 2.0.0.0/8

# Allow ICMP between jail1 and correct subnet, I think this is where the problem!!
$cmd 0050 allow icmp from 2.0.0.0/8 to 10.0.0.1 out via $int_if1 $ks
$cmd 0051 allow icmp from 10.0.0.1 to 2.0.0.0/8 in via $int_if1 $ks

#Deny Everything else by default
$cmd 0999 deny log all from any to any
```


My ipfw.rules on the jail (I call it jailfw.rules)

```
####################################
###########    JAIL 1	############
####################################

#!/bin/sh
# Start of IPFW Rules File
ipfw -q -f flush		# Flush rules

# Set defaults
ks="keep-state"
cmd="ipfw -q add"		# Add new rule
pif="epair0b"			# Main interface name
int_if="eoaur2b"		# Private interface name

# No restriction on loopback
$cmd 0005 allow all from any to any via lo0

# Allow packet through if previously added
$cmd 0010 check-state

# Allow incoming and outgoing ping to the main interface
$cmd 015 allow icmp from any to any out via $pif $ks
$cmd 016 allow icmp from any to any in via $pif $ks


# Allow ICMP from friendly subnet
$cmd 0030 allow icmp from 10.0.0.1 to any out via $int_if $ks
$cmd 0031 allow icmp from any to 10.0.0.1 in via $int_if $ks

#Deny Everything else by default
$cmd 0999 deny log all from any to any
```



All the interfaces on the jails are virtual and are destroyed with the jails are destroyed (on system shutdown).


----------

