# 7 NIC - router/firewall with PF  - no rdr on one interface



## TheSkunk (Mar 6, 2013)

Hello,

I've got a PC with 7 NICs (rl0, vr0, re0-re4 - see please pf.conf for details) which I use as a router and firewall to connect 2 LANs a DMZ via 3 gateways to the internet. I'm using FreeBSD 9.1 and pf (current version). So far, so good - everything works fine. I use 3 NICs for 3 different gateways (2 with fixed IPs, one dynamic) to the internet. From 2 LANs I could balance and queue traffic between those 3 gateways (gw0-gw2), and I'm also able to rdr - BUT, and that's the point, not on all NICs, just on one. The gw0 works, the gw1 not (gw2 is not used for rdr, because of the dynamic IP). There is definitely no reaction on that interface / NIC if you try to access it from outside (tried everything, nmap als means the host is probably down). But, traffic to the outside, over that interface works fine via NAT.

My answer to that is: "Bug in pf, bug in FreeBSD ..." or do you have a different opinion, some other options, etc? So please help me in that case, thank you 


ps. 

I did all my homework 
everything is set and activated like it should (rc.conf, sysctl.conf, rc.local...), all NICs are up and working fine (with) and without pf enabled. Else, rdr wouldn't work with gw0 too ;-) ...
I checked everything outside the wall (that means, the router from my provider, which is connected to gw1 works fine. If I connect an other device, like f.e. a Zywall 2, it will work like it should (you can access server B)
DMZ is not in use - Server A and B are located in the LAN because of development purpose
The problem seems to be here: 
	
	



```
rdr pass on $gw1_if proto tcp from any to any port { 80, 22 } -> $srvB
```
 - but this rule does not really differ from the gw0 rule
It makes no difference if you delete all filter rules and use just 
	
	



```
pass in all
pass out all
```
 - there will be still no reaction on gw1 if you rdr
Here is my pf.conf:


```
gw0_if="re4"
gw1_if="rl0"
gw2_if="vr0"
lan1_if="re3"
lan2_if="re2"
dmz_if="re1"

gw0_net="10.0.0.0/24"
gw1_net="10.0.1.0/24"
gw2_net="10.0.2.0/24"
lan1_net="192.168.0.0/24"
lan2_net="192.168.1.0/24"
dmz_net="192.168.2.0/24"

gw0="10.0.0.138"
gw1="10.0.1.138"
gw2="10.0.2.138"
lan1="192.168.0.1"
lan2="192.168.1.1"
dmz="192.168.2.1"

srvA="192.168.0.54"
srvB="192.168.0.50"


table <lan1_trusted> const { 192.168.0. ...}
table <lan1_prio> const { 192.168.0. ...}
table <gw0_trusted> const { 192.168.0. ...}
table <gw0_mandatory> const { 192.168.0. ...}
table <gw1_trusted> const { 192.168.0. ...}
table <gw0_routed> const { a.b.c.d ...}
table <gw1_routed> const { a.b.c.d ...}
table <gw_admin> const { 192.168.0. ... }
table <lan1_master> const { 192.168.0. ... }
table <ssh_trusted> const { 192.168.0. ... a.b.c.d ...}
table <peter> const { 80.109.197.142 }

t_p = "{tcp, udp, icmp}"
d_p = "{tcp, udp}"


# 3) Queue
# --------------- GW0 (nach Internet auf 8/1 DIC)
altq on $gw0_if cbq bandwidth 960Kb queue { gw0_def, gw0_lan1, gw0_dmz}
queue gw0_def bandwidth 70% cbq(default)
queue gw0_lan1 bandwidth 10% { gw0_lan1_pri, gw0_lan1_oth }
  queue gw0_lan1_pri bandwidth 70% priority 4 cbq(red borrow)
  queue gw0_lan1_oth bandwidth 30% priority 1 cbq(red borrow)
queue gw0_dmz bandwidth 20% priority 6 cbq(red borrow)
# --------------- GW1 (nach Internet auf 16/1 DIC)
altq on $gw1_if cbq bandwidth 4Mb queue { gw1_def, gw1_lan1, gw1_lan2, gw1_dmz}
queue gw1_def bandwidth 50% cbq(default)
queue gw1_lan1 bandwidth 20% { gw1_lan1_pri, gw1_lan1_oth }
  queue gw1_lan1_pri bandwidth 70% priority 6 cbq(red borrow)
  queue gw1_lan1_oth bandwidth 30% priority 3 cbq(red borrow)
queue gw1_lan2 bandwidth 10% priority 1 cbq(red borrow)
queue gw1_dmz bandwidth 20% priority 4 cbq(red borrow)
# --------------- GW2 (nach Internet auf 16/1 Tele2)
altq on $gw2_if cbq bandwidth 960Kb queue { gw2_def, gw2_lan1, gw2_lan2}
queue gw2_def bandwidth 70% cbq(default)
queue gw2_lan1 bandwidth 15% { gw2_lan1_pri, gw2_lan1_oth }
  queue gw2_lan1_pri bandwidth 60% priority 3 cbq(red borrow)
  queue gw2_lan1_oth bandwidth 40% priority 1 cbq(red borrow)
queue gw2_lan2 bandwidth 15% priority 2 cbq(red borrow)


# 4) NAT
# --------------- lan1 -> Internet
nat on $gw0_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw1_net, !$gw2_net } -> ($gw0_if:0)
nat on $gw1_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw0_net, !$gw2_net } -> ($gw1_if:0)
nat on $gw2_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw1_net, !$gw0_net } -> ($gw2_if:0)
# --------------- lan2 -> Internet
nat on $gw1_if from $lan2_if:network to { any, !$dmz_net } -> ($gw1_if:0)
nat on $gw2_if from $lan2_if:network to { any, !$dmz_net } -> ($gw2_if:0)
# --------------- DMZ -> Internet
nat on $gw0_if from $dmz_if:network to any -> ($gw0_if:0)
nat on $gw1_if from $dmz_if:network to any -> ($gw1_if:0)


# 5) RDR
# --------------- srvA - production
rdr pass on $gw0_if proto tcp from any to any port { 25, 465, 143, 993, 587, 80, 443, 22 }  -> $srvA
# --------------- srvB - development
rdr pass on $gw1_if proto tcp from any to any port { 80, 22 } -> $srvB


# 6) 
pass in quick from <lan1_trusted> to $lan1
antispoof quick for { lo $gw0_if $gw1_if $gw2_if $lan2_if}


# 7) for Loadbalan1cing & Queuing
pass out quick on $gw0_if from <gw0_mandatory> keep state queue(gw0_def, gw0_dmz)
# --------------- 
pass out quick on $gw2_if from <lan1_prio> keep state queue(gw2_lan1_pri, gw2_def)
pass out quick on $gw2_if from $lan1_net keep state queue(gw2_lan1_oth, gw2_def)
pass out quick on $gw2_if from $lan2_net keep state queue(gw2_lan2, gw2_def)
pass out quick on $gw1_if from <lan1_prio> keep state queue(gw1_def, gw1_lan1_pri)
pass out quick on $gw1_if from $lan1_net keep state queue(gw1_def, gw1_lan1_oth)
pass out quick on $gw1_if from $dmz_net keep state queue(gw1_def, gw1_dmz)
pass out quick on $gw0_if from <lan1_prio> keep state queue(gw0_def, gw0_lan1_pri)
pass out quick on $gw0_if from $lan1_net keep state queue(gw0_def, gw0_lan1_oth)
pass out quick on $gw0_if from $dmz_net keep state queue(gw0_def, gw0_dmz)
# --------------- 
pass out on $gw0_if route-to ($gw1_if $gw1) from $gw1_if
pass out on $gw1_if route-to ($gw0_if $gw0) from $gw0_if
pass out on $gw1_if route-to ($gw2_if $gw2) from $gw2_if
pass out on $gw2_if route-to ($gw1_if $gw1) from $gw1_if


# 8) special traffic
# --------------- lan1 --> DMZ and lan2
pass in quick on $lan1_if from $lan1_net to $dmz_if keep state
pass in quick on $lan1_if from $lan1_net to $lan2_if keep state
# --------------- lan1 --> special
pass in quick on $lan1_if from <gw0_trusted> to <gw0_routed> keep state rtable 0
pass in quick on $lan1_if from <gw_admin> to $gw0_net keep state rtable 0
pass in quick on $lan1_if from <gw1_trusted> to <gw1_routed> keep state rtable 1
pass in quick on $lan1_if from <gw_admin> to $gw1_net keep state rtable 1
pass in quick on $lan1_if from <gw_admin> to $gw2_net keep state rtable 2
# --------------- lan1 --> Internet
pass in quick on $lan1_if route-to { ($gw1_if $gw1), ($gw2_if $gw2), ($gw0_if $gw0) } round-robin from <lan1_prio> to any keep state
pass in quick on $lan1_if route-to { ($gw1_if $gw1), ($gw2_if $gw2) } round-robin from $lan1_net to any keep state
# --------------- lan2 --> Internet
block in quick on $lan2_if from any to $gw0_if
pass in quick on $lan2_if route-to { ($gw2_if $gw2), ($gw1_if $gw1) } round-robin from $lan2_net to any keep state


# 9) IN
# --------------- GW0 -> srvA
pass in quick on $gw0_if proto TCP from any to $srvA port { 25, 465, 143, 993, 587, 80, 22 } keep state rtable 0
# --------------- GW1 (vi lan1) -> srvB
pass in quick on $gw1_if proto TCP from any to $srvB port { 22, 80 } keep state rtable 1


# 10) BLOCK
block in on $gw0_if all
block in on $gw1_if all
block in on $gw2_if all
block in on $lan2_if all
```


`# setfib 0 netstat -nr -f inet` produces this 
	
	



```
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.0.0.138         UGS         0      167    re4
10.0.0.0/24        link#5             U           0     5334    re4
10.0.0.1           link#5             UHS         0        2    lo0
10.0.1.0/24        link#7             U           0     1019    rl0
10.0.1.1           link#7             UHS         0        0    lo0
10.0.2.0/24        link#6             U           0       14    vr0
10.0.2.1           link#6             UHS         0        0    lo0
127.0.0.1          link#10            UH          0      676    lo0
192.168.0.0/24     link#4             U           0  3719388    re3
192.168.0.1        link#4             UHS         0        2    lo0
192.168.1.0/24     link#3             U           0  4066489    re2
192.168.1.1        link#3             UHS         0        0    lo0
192.168.2.0/24     link#2             U           0        0    re1
192.168.2.1        link#2             UHS         0        0    lo0
```
 and `setfib 1` and `2` the same with different gates/default routes (10.0.1.138 and 10.0.2.138)


----------



## TheSkunk (Mar 6, 2013)

I did some testings with a laptop on the gw1 interface. I connected to it directly via a CAT5 cable, so I could ping, login, etc. to the firewall (opened the needed ports first, made sshd available on that interface), but I still couldn't redirect my packages through it... so there seems to be really a bug.


----------



## kpa (Mar 6, 2013)

Try these, one after another and see if either makes a difference:

`# ifconfig rl0 -rxcsum`

`# ifconfig rl0 -txcsum`


----------



## J65nko (Mar 7, 2013)

TheSkunk said:
			
		

> I did some testings with a laptop on the gw1 interface. I connected to it directly via a CAT5 cable, so I could ping, login, etc. to the firewall (opened the needed ports first, made sshd available on that interface), but I still couldn't redirect my packages through it... so there seems to be really a bug.



Could you retry this procedure with the following pf.conf?:


```
gw1_if = lo3
srvB = "192.168.0.50"

#rdr on $gw1_if proto tcp from any to any port { 80, 22 } tag SERVER_B  -> $srvB
rdr log (all) on $gw1_if proto tcp from any to any port { 80 } [color=blue]tag SERVER_B[/color]  -> $srvB

# for SSH login and debugging
pass in quick on  $gw1_if inet proto tcp to $gw1_if port ssh

pass in  log (all) quick on $gw1_if  [color=blue]tagged SERVER_B[/color]
pass out log (all) quick on $lan1    [color=blue]tagged SERVER_B[/color]

block log all
```
To do a syntax test for this pf on a single NIC machine, I had to cheat by creating a fake lo3 interface. So you have to change that to the "real" one 

The basic idea is to add a tag or label to the redirected packets and follow them while being processed by pf.

To test you will need to ssh(1) from your laptop to your router using a couple of xterms.

In the first xterm you run tcpdump(1) on the external interface of your router:
`# tcpdump -tttt -ni rl0 port 80`

You monitor the internal interface in another one with:
`# tcpdump -tttt -ni re0 port 80`

In a third xterm you check the pflog0 device
`# tcpdump -eni pflog0`. 

This way you can check if the redirection or translation of the destination address is being done or not, whether the incoming translated packets on the external and internal interfaces are passed by pf.


----------



## TheSkunk (Mar 7, 2013)

Than*ks* guys for the help, but *I* think I was wrong with my all statements. I had a very ugly behavior on my router in front of rl0. The first two or three times everything worked fine, than I put the router on rl0 and the router changed its behavior (it*'*s especially branded by our Internet provider, they renew the firmware every couple of days, this without any warning - assh....). This router/modem (technicolor 788A1vn) blocked from now on every traffic from the outside to the inside which was not stated, but with port-forwarding properly set and a disabled firewall function in the GUI! I did my last test before the firmeware update, so I thought it must be on my rl0 (gw1), and another test in the meantime went probably wrong (my fault, sorry) - but, in total I was wrong. Sorry for that, I thing I've got to talk to my provider (uuuhhh I'm pissed off now ... )

Than*ks* you everyone and sorry for that thread I opened (will say, it's SOLVED now).


----------



## TheSkunk (Mar 7, 2013)

Hello again,

I'm sorry, but I hope we could reopen this thread again, because I've got still these problems (but I've also some new information for you ;-) ). 

What happens in the meantime: 

After a nice discussion with my provider, now the internet line is working fine, traffic from outside and inside is routed. After some tests with a Zywall 2, I connected with my firewall/router again - and the same problems occur again. As I've already written above, I managed my 3 default gateways via rtables in pf (ok, I tried it ;-)). I can see and use those 3 gateways with `setfib 0[,1,2]`. Everything works if I just use the interface with disabled pf. I also could run 3 SSH daemons on different gateways, that is also working  perfect, I could connect over all gateways to the firewall via ssh (I know, this is not secure - but I just used it for testing). 

Now, when I'm turning pf on, NAT is working, rdr on re4 (that is $gw0_if) too, but on rl0 ($gw1_if) nothing seems to happen when traffic comes in. I monitored these situation - as you suggested - on these two interfaced and watched what was happening when I connect via a special host from outside via ssh, redirecting to one host inside the LAN. If I come in over the interface which also represents the default gateway via rtable 0 ($gw0_if, re4) and which is the global default gateway for everything else, then everything works fine. If I come in via rtable 1 ($gw1_if, rl0), I can see the request on the host and on the gateway interface I monitored. But I can also see, that my default gateway (re4) is answering with the IP of rl0 (rl0 (the host) by itself is not answering). It seems, that my firewall is spoofing a little bit, but I've got no idea what to do now. I also tried to put every interface in the right rdomain to activate rtable for it, but that seems not to work (it's described here: http://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/). 

I put route-to options in the pass in lines in pf.conf, I tried to block the traffic if it comes from the wrong interface with the wrong IP, etc. etc. Only the default gateway (this on rtable 0) is still answering, nobody else. (ps. does somebody know, how to put the rdomain directive into the interface declaration line in pf.conf? - could that be a problem?)

Best regards, and thank you in advance for helping.


----------



## J65nko (Mar 8, 2013)

You mention that the firewall seems to be spoofing. If you compare the arp(4) tables for each routing domain, does that give any clue?

Is it possible to assign the public fixed IP addresses to your FreeBSD firewall box without intermediation of that troublesome ISP router?


----------



## TheSkunk (Mar 8, 2013)

No, I'm sorry, those routers are locked by the provider (they have got their own firmware). I can just configure portforwarding on them - the outside fixed IP couldn't be routed. But that works all, that shouldn't bee the problem anymore. 

Well, because of the fact, that rdr is not working like it should (and/or rtable is not working like it should), I tried to use your example from above, but instead of rdr with the [CMD=""]route-to[/CMD] option. I also target now just one server (srvA) from two different gateways (easier testing, same conditions).


```
# 1) Definitions  -------------------------------------------------------------------------------------------
gw0_if="re4"
gw1_if="rl0"
gw2_if="vr0"
lan1_if="re3"
lan2_if="re2"
dmz_if="re1"

gw0_net="10.0.0.0/24"
gw1_net="10.0.1.0/24"
gw2_net="10.0.2.0/24"
lan1_net="192.168.0.0/24"
lan2_net="192.168.1.0/24"
dmz_net="192.168.2.0/24"

gw0="10.0.0.138"
gw1="10.0.1.138"
gw2="10.0.2.138"
lan1="192.168.0.1"
lan2="192.168.1.1"
dmz="192.168.2.1"

srvA="192.168.0.54"
srvB="192.168.0.50"

table <lan1_trusted> const { 192.168.0.... }
table <lan1_prio> const { 192.168.0.... }
table <gw0_trusted> const { 192.168.0.... }
table <gw0_mandatory> const { 192.168.0.50 192.168.0.54 }
table <gw1_trusted> const { 192.168.0.... }
table <gw0_routed> const { ... }
table <gw1_routed> const { 83.218.163.44 }
table <gw_admin> const { 192.168.0.... }
table <lan1_master> const { 192.168.0... }
table <xs_trusted> const { 192.168.0.54 192.168.0.50 ...}
table <ssh_trusted> const { 192.168.0.54 192.168.0.50 ...}

t_p = "{tcp, udp, icmp}"
d_p = "{tcp, udp}"

# 2) Options -------------------------------------------------------------------------------------------------
set block-policy drop
set limit { frags 50000, states 10000, src-nodes 100000 }

# 3) Speedlimiting / Queuing (only OUT) ----------------------------------------------------------------------
# --------------- GW0 (-> Internet 8/1 DIC, ADSL, 81.16.104.18)
altq on $gw0_if cbq bandwidth 960Kb queue { gw0_def, gw0_lan1, gw0_dmz}
queue gw0_def bandwidth 70% cbq(default)
queue gw0_lan1 bandwidth 10% priority 5 cbq(red borrow) { gw0_lan1_pri, gw0_lan1_oth }
  queue gw0_lan1_pri bandwidth 70% priority 4 cbq(red borrow)
  queue gw0_lan1_oth bandwidth 30% priority 1 cbq(red borrow)
queue gw0_dmz bandwidth 20% priority 6 cbq(red borrow)
# --------------- GW1 (-> Internet 30/4 DIC, VDSL2, 81.16.104.142)
altq on $gw1_if cbq bandwidth 4Mb queue { gw1_def, gw1_lan1, gw1_lan2, gw1_dmz}
queue gw1_def bandwidth 30% cbq(default)
queue gw1_lan1 bandwidth 40% priority 6 cbq(red borrow) { gw1_lan1_pri, gw1_lan1_oth }
  queue gw1_lan1_pri bandwidth 70% priority 5 cbq(red borrow)
  queue gw1_lan1_oth bandwidth 30% priority 3 cbq(red borrow)
queue gw1_lan2 bandwidth 10% priority 1 cbq(red borrow)
queue gw1_dmz bandwidth 20% priority 4 cbq(red borrow)
# --------------- GW2 (-> Internet 16/1 Tele2, xDSL, dyn IP)
altq on $gw2_if cbq bandwidth 960Kb queue { gw2_def, gw2_lan1, gw2_lan2}
queue gw2_def bandwidth 40% cbq(default)
queue gw2_lan1 bandwidth 30% priority 6 cbq(red borrow) { gw2_lan1_pri, gw2_lan1_oth }
  queue gw2_lan1_pri bandwidth 60% priority 3 cbq(red borrow)
  queue gw2_lan1_oth bandwidth 40% priority 1 cbq(red borrow)
queue gw2_lan2 bandwidth 30% priority 2 cbq(red borrow)

# 4) NAT ------------------------------------------------------------------------------------------------------
# --------------- lan1 -> Internet
nat on $gw1_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw0_net, !$gw2_net } tag NAT_GW1_LAN1 -> ($gw1_if:0)
nat on $gw2_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw1_net, !$gw0_net } tag NAT_GW2_LAN1 -> ($gw2_if:0)
nat on $gw0_if from $lan1_if:network to { any, !$lan1_net, !$dmz_net, !$gw1_net, !$gw2_net } tag NAT_GW0_LAN1 -> ($gw0_if:0)
# --------------- lan2 -> Internet
nat on $gw2_if from $lan2_if:network to { any, !$dmz_net } tag NAT_GW2_LAN2 -> ($gw2_if:0)

# 5) RDR ------------------------------------------------------------------------------------------------------
# --------------- srvA (srvB not in use)
rdr pass on $gw0_if proto tcp from any to any port { 59653, 25, 465, 143, 993, 587, 80, 443 } tag RDR_GW0_srvA -> $srvA
# following rdr rules are disabled for testing, because of the use of route-to ...
#rdr pass on $gw0_if proto tcp from any to any port { 45622 } tag RDR_SSH_GW0_srvA -> $srvA
#rdr pass on $gw1_if proto tcp from any to any port { 45622 } tag RDR_SSH_GW1_srvA -> $srvA

# 6) vom System und Admin alles zulassen ----------------------------------------------------------------------
pass in quick from <lan1_trusted> to $lan1

# 7) IN -------------------------------------------------------------------------------------------------------
# --------------- GW0 -> srvA
pass in log (all) quick on $gw0_if proto TCP from <xs_trusted> to $srvA port 59653 keep state
pass in log (all) quick on $gw0_if proto TCP from any to $srvA port { 25, 465, 143, 993, 587, 80 } keep state
pass in log (all) quick on $gw0_if route-to ($lan1_if $lan1) proto TCP from <ssh_trusted> to $srvA port 45622 tag RDR_SSH_GW0_srvA keep state
pass in log (all) quick on $gw1_if route-to ($lan1_if $lan1) proto TCP from <ssh_trusted> to $srvA port 45622 tag RDR_SSH_GW1_srvA keep state

# 8) OUT - Out, Loadbalancing & Queuing ----------------------------------------------------------------------------
pass out quick on $lan1_if tagged RDR_SSH_GW0_srvA
pass out quick on $lan1_if tagged RDR_SSH_GW1_srvA
# --------------- Loadbalancing
pass out on $gw0_if route-to ($gw1_if $gw1) from $gw1_if
pass out on $gw1_if route-to ($gw0_if $gw0) from $gw0_if
pass out on $gw1_if route-to ($gw2_if $gw2) from $gw2_if
pass out on $gw2_if route-to ($gw1_if $gw1) from $gw1_if
# --------------- Queuing
pass out quick on $gw2_if from <lan1_prio> keep state queue(gw2_lan1_pri, gw2_def)
pass out quick on $gw2_if from $lan1_net keep state queue(gw2_lan1_oth, gw2_def)
pass out quick on $gw2_if from $lan2_net keep state queue(gw2_lan2, gw2_def)
pass out quick on $gw1_if from <lan1_prio> keep state queue(gw1_def, gw1_lan1_pri)
pass out quick on $gw1_if from $lan1_net keep state queue(gw1_def, gw1_lan1_oth)
pass out quick on $gw1_if from $dmz_net keep state queue(gw1_def, gw1_dmz)
pass out quick on $gw0_if from <lan1_prio> keep state queue(gw0_def, gw0_lan1_pri)
pass out quick on $gw0_if from $lan1_net keep state queue(gw0_def, gw0_lan1_oth)
pass out quick on $gw0_if from $dmz_net keep state queue(gw0_def, gw0_dmz)

# 9) OUT - filtered & routed traffic --------------------------------------------------------------------------
# --------------- LAN1 -> DMZ & LAN2
pass in quick on $lan1_if from $lan1_net to $dmz_if keep state tag LAN1_GW1_DMZ
pass in quick on $lan1_if from $lan1_net to $lan2_if keep state tag LAN1_GW1_LAN2
# --------------- LAN1 - special
pass in quick on $lan1_if route-to { ($gw0_if $gw0) } round-robin from <gw0_trusted> to <gw0_routed> tag LAN1_GW0_GW0 keep state
pass in quick on $lan1_if route-to { ($gw0_if $gw0) } round-robin from <gw_admin> to $gw0_net tag LAN1_GW0 keep state
pass in quick on $lan1_if route-to { ($gw1_if $gw1) } round-robin from <gw1_trusted> to <gw1_routed> tag LAN1_GW1_GW1 keep state
pass in quick on $lan1_if route-to { ($gw1_if $gw1) } round-robin from <gw_admin> to $gw1_net tag LAN1_GW1 keep state
pass in quick on $lan1_if route-to { ($gw2_if $gw2) } round-robin from <gw_admin> to $gw2_net tag LAN1_GW2 keep state
# --------------- LAN1 -> Internet
pass in quick on $lan1_if route-to { ($gw1_if $gw1), ($gw2_if $gw2) } round-robin from <lan1_prio> to any tag LAN1_RT_GW1_GW2_PRIO keep state
pass in quick on $lan1_if route-to { ($gw2_if $gw2) } round-robin from $lan1_net to any tag LAN1_RT_GW2_GW1 keep state
# --------------- LAN2 -> Internet
block in quick on $lan2_if from any to $gw0_if
pass in quick on $lan2_if route-to { ($gw2_if $gw2) } round-robin from $lan2_net to any tag LAN2_RT_GW2 keep state

# 10) block ----------------------------------------------------------------------------------------------------
block in on $gw0_if all
block in on $gw1_if all
block in on $gw2_if all
block in on $lan2_if all
```


... yes, and that all is also not working properly (route-to works from LAN to the Internet, but not on my gateways?!). On the two xterms, where I monitor the traffic, I could see the requests, but there is no answer passed through. The xterm for pflog0 stays quiet too. As you see, states should be kept, etc. Again, if I use rdr from your example, it is spoofing - and/because the default gateway ($gw0_if, re4) is always used.

Should I try to install OpenBSD, or might there be a solution on FreeBSD? (Then I've got to recompile the driver of two NICs too ... it's frustrating ...  )


----------



## J65nko (Mar 8, 2013)

Because of the spoofing I asked you to compare the arp(4) tables, but you did not 

`# setfib 0 arp -an ; setfib 1 arp -an; setfib 2 arp -an`

Does this show anything interesting?

With fixed IP addresses, as far as I understand  you should be able to do without different routing domains and use route-to.

RE: OpenBSD

You could try OpenBSD. Please remember that OpenBSD pf has moved on to a newer syntax for NAT and redirection. See http://www.openbsd.org/faq/pf/rdr.html

On FreeBSD pf is being adjusted to make use of multiple CPU cores, while on OpenBSD only 1 single core will be used. x(

But with OpenBSD you could approach Joel Knight at http://www.packetmischief.ca for assistance.
Choices and trade-offs, life is full of them


----------



## TheSkunk (Mar 11, 2013)

Hi, it seems everything is ok, in all three result tables all records are identical. I will not mess around longer with that problem on FreeBSD. I'll try to check it out with an other HDD where I will install OpenBSD. Then I will let you know, and thank you all for helping until now ;-)


----------

