# Using setfib with same gateway on both interfaces?



## ebichu (Jan 23, 2011)

I have 3 physical interfaces (msk0, sk0, xl0) in one FreeBSD 8.1-box which is used as my gateway with PF-NAT.
The thing I want is to isolate one of my jails to xl0. My ISP gives me 5 dynamic public IP's. 
I've tried seting up multiple routes with setfib without success and I think the problem lies with the ISP giving me the same gateway on every physical interface I try with dhcp.

Is that a problem?

uname -a:

```
FreeBSD xxx.xxx.xxx 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Thu Jan 20 17:57:48 UTC 2011 root@xxx.xxx.xxx:/usr/src/sys/amd64/compile/lain  amd64
```
ifconfig:

```
msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
	ether X:X:X:X:X
	inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
	inet 10.0.1.10 netmask 0xffffffff broadcast 10.0.1.10
	inet 10.0.1.20 netmask 0xffffffff broadcast 10.0.1.20
	inet 10.0.1.30 netmask 0xffffffff broadcast 10.0.1.30
	media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
	status: active
xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
	ether X:X:X:X:X
	inet A.229.66.183 netmask 0xfffff000 broadcast A.229.79.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
	ether X:X:X:X:X
	inet A.229.70.4 netmask 0xfffff000 broadcast A.229.79.255
	media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
	status: active
```

setfib 0 netstat -rn:

```
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            A.229.64.1        UGS         0  1941879    sk0
10.0.1.0/24        link#1             U          26  4392385   msk0
10.0.1.1           link#1             UHS         0        0    lo0
10.0.1.10          link#1             UHS         0        0    lo0 =>
10.0.1.10/32       link#1             U           0        0   msk0
10.0.1.20          link#1             UHS         0        0    lo0 =>
10.0.1.20/32       link#1             U           0        0   msk0
10.0.1.30          link#1             UHS         0       33    lo0 =>
10.0.1.30/32       link#1             U           0        0   msk0
A.229.64.0/20     link#3             U           0        0    sk0
A.229.66.183      link#2             UHS         0        3    lo0
A.229.70.4        link#3             UHS         0       16    lo0
127.0.0.1          link#4             UH          0     4828    lo0
```

setfib 1 netstat -rn
	
	



```
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            A.229.64.1        UGS         0      165    xl0
10.0.1.0/24        link#1             U           0        0   msk0
10.0.1.10/32       link#1             U           0        0   msk0
10.0.1.20/32       link#1             U           0        0   msk0
10.0.1.30/32       link#1             U           0        0   msk0
A.229.64.0/20     link#3             U           0        1    sk0
127.0.0.1          link#4             UH          0        0    lo0
```

pf.conf
	
	



```
# macros
ext_if="sk0"
int_if="msk0"
# options
set loginterface none
set block-policy drop
set state-policy if-bound
set debug none
set skip on lo0
scrub in

# Usual NAT
nat on $ext_if from !($ext_if) -> ($ext_if:0)

rdr on $ext_if proto tcp from any to ($ext_if) port smtp -> 10.0.1.1 port smtp

# filter rules
block in
pass out quick
pass in quick proto tcp from any to any port 25
pass in quick on $int_if

pass in quick on xl0 \
        reply-to ( xl0 A.229.64.1 ) \
        proto tcp from any to any port { 8010 } \
        keep state ( floating )
```


----------



## SirDice (Jan 25, 2011)

Both sk0 and xl0 are on the same subnet. Routing will not separate the traffic.


----------



## ebichu (Jan 25, 2011)

Oh. Do you have any clue as to why it behaves like this? 
I forgot to tell how the gateway is connected to the internet.
Interface sk0 is connected directly to one of 4 ports of a VDSL2-modem (60/20Mbit)  
and msk0 is connected to a 3com Officeconnect 100Mbit switch. Interface xl0 is connected directly to the modem.

Could that be the problem?


I mean if I have 5 public IP's at my disposal from my ISP the traffic ought to be isolated 
if I'm connecting another computer directly to the VDSL2-modem. Otherwise it's no point having all those IP's.


----------



## SirDice (Jan 25, 2011)

ebichu said:
			
		

> Oh. Do you have any clue as to why it behaves like this?


Ehm, yes. It's basic TCP/IP actually :stud

But perhaps you and I have a different meaning to "isolating"? Perhaps you could explain more what you mean by that?

You normally bind a jail to a specific interface with a specific IP address. All network services on both the host and the jail are also configured to use one or more specific IP addresses.



> I forgot to tell how the gateway is connected to the internet.
> Interface sk0 is connected directly to one of 4 ports of a VDSL2-modem (60/20Mbit)
> and msk0 is connected to a 3com Officeconnect 100Mbit switch. Interface xl0 is connected directly to the modem.
> 
> Could that be the problem?


I don't really see a problem.



> I mean if I have 5 public IP's at my disposal from my ISP the traffic ought to be isolated if I'm connecting another computer directly to the VDSL2-modem.


No, not if all those IP addresses are on the same subnet. Certainly not if they all come from the same modem.



> Otherwise it's no point having all those IP's.


That depends on what you planned on doing with them.


----------



## ebichu (Jan 25, 2011)

SirDice said:
			
		

> Ehm, yes. It's basic TCP/IP actually :stud
> 
> But perhaps you and I have a different meaning to "isolating"? Perhaps you could explain more what you mean by that?
> 
> ...



Yes like that. For example if the jail is bound to xl0 and I start httpd in that specific jail both incoming and outgoing traffic should be isolated in that jail and on that interface. 



> I don't really see a problem.


Thought so too but was unsure.




> No, not if all those IP addresses are on the same subnet. Certainly not if they all come from the same modem.



I just tested connecting one of my other boxes (OS X 10.6.6) to port number 2 of the VDSL2-modem and came up with the following information:


```
[B]ifconfig[/B]
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether X:X:X:X:X 
	inet A.226.168.184 netmask 0xfffffc00 broadcast A.226.171.255
	media: autoselect
	status: active
```


```
[B]netstat -rn[/B]
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            A.226.168.1        UGSc           46        0     en0
A.226.168/22       link#4             UCS             4        0     en0
A.226.168.1        0:d0:2:c8:7c:0     UHLWI          29        0     en0   1047
A.226.168.184      127.0.0.1          UHS             0        0     lo0
A.226.171.255      link#4             UHLWbI          3      134     en0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              0        0     lo0
169.254            link#4             UCS             0        0     en0
```
That depends on what you planned on doing with them.


----------



## SirDice (Jan 26, 2011)

ebichu said:
			
		

> Yes like that. For example if the jail is bound to xl0 and I start httpd in that specific jail both incoming and outgoing traffic should be isolated in that jail and on that interface.


The traffic isn't isolated. It only gets send to/from the jail's services. But you need to make sure those services are bound to the correct IP address and not to all (usually indicated by * or 0.0.0.0). That's the normal way of operating a jail, there's no need fiddle with multiple routing tables.


----------



## ebichu (Jan 27, 2011)

Maybe I'm using the wrong definition of "isolated"?
Just want the traffic to be seen as coming from xl0 and the dynamic public ip that I get on that interface.

Shouldn't it work with xl0 being configured to get a dynamic public ip, 
putting the jail on a private ip and then putting some rdr pass rule and pass out in pf.conf?

Another possibility would be that xl0 doesn't get a public ip on a separate subnet when requested by dhclient 
which perhaps prevents what I'm trying to do working... 
My other box got an IP on another "correct" subnet when connected to my VDSL2-modem.

I'm sorry for my confused post...


----------

