# PF load balancing



## dextro_ (Oct 17, 2009)

I wanted to setup a machine as my LAN gateway and have it load balance over multiple WANs. When I found http://www.openbsd.org/faq/pf/pools.html I choose FreeBSD as the machines OS. After getting it up and running, and acting as a gateway just using one WAN via 
	
	



```
# macros
wan1="nfe0"
lan1="rl0"

pc1="10.0.0.2"
xb1="10.0.0.3"

# options
#set block-policy return
#set loginterface $wan1
set skip on lo

# scrub
#scrub in

# nat/rdr
nat on $wan1 from !($wan1) -> ($wan1:0) static-port
#rdr-anchor "miniupnpd"

# uTorrent
rdr on $wan1 proto tcp from any to any port 41016 -> $pc1

# Apache
#rdr on $wan1 proto tcp from any to any port 1337 -> $pc1

# Remote Desktop
#rdr on $wan1 proto tcp from any to any port 3389 -> $pc1

# NetMeeting
#rdr on $wan1 proto {tcp, udp} from any to any port 1:65535 -> $pc1

# Hamachi
#rdr on $wan1 proto {tcp, udp} from any to any port 39004 -> $pc1

# Tiberian Sun / Red Alert 2 / Yuri's Revenge
#rdr on $wan1 proto {tcp, udp} from any to any port 1234 -> $pc1

# Xbox Live
rdr on $wan1 proto {tcp, udp} from any to any port 3074 -> $xb1

#anchor "miniupnpd"
```
 I decided to try the load balancing and came up with 
	
	



```
lan1r = "10.0.0.0/24"
lan1  = "rl0"
wan1 = "nfe0"
wan2 = "rl1"
gw1 = "10.0.1.2"
gw2 = "10.0.2.2"

# nat outgoing connections on each internet interface
#nat on $ext_if1 from $lan_net to any -> ($ext_if1)
#nat on $ext_if2 from $lan_net to any -> ($ext_if2)
nat on $wan1 from $lan1r to any -> ($wan1) #static-port
nat on $wan2 from $lan1r to any -> ($wan2) #static-port

# default deny
block in from any to any
block out from any to any

# pass all outgoing packets on internal interface
pass out on $lan1 from any to $lan1r

# pass in quick any packets destined for the gateway itself
pass in quick on $lan1 from $lan1r to $lan1

# load balance outgoing tcp traffic from internal network.
pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto tcp from $lan1r to any flags S/SA modulate state

# load balance outgoing udp and icmp traffic from internal network
pass in on $lan1 route-to { ($wan1 $gw1), ($wan2 $gw2) } round-robin proto { udp, icmp } from $lan1r to any keep state

# general "pass out" rules for external interfaces
pass out on $wan1 proto tcp from any to any flags S/SA modulate state
pass out on $wan1 proto { udp, icmp } from any to any keep state
pass out on $wan2 proto tcp from any to any flags S/SA modulate state
pass out on $wan2 proto { udp, icmp } from any to any keep state

# route packets from any IPs on $ext_if1 to $ext_gw1 and the same for $ext_if2 and $ext_gw2
pass out on $wan1 route-to ($wan2 $gw2) from $wan2 to any
pass out on $wan2 route-to ($wan1 $gw1) from $wan1 to any
```
 which does nothing aparently, when I pfctl -f /etc/pf.lb.conf no errors are reported and my LAN looses internet connectivity.

Does any one see the problem? I can ping Google fine using either WAN as default route so it has to be my PF conf.


----------



## vivek (Oct 19, 2009)

If I were you I will go with pfsense for load balancing. It offers modified FreeBSD + PF with web based user interface.


----------



## dextro_ (Oct 19, 2009)

I doubt they modified PF in any way and all they did was strip FreeBSD and make it useless. The machine has many other functions that pfSense limits me from doing, like compiling. Thanks for the suggestion though.


----------



## dextro_ (Oct 21, 2009)

Bump!

In /etc/rc.conf I have


```
gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.lb2.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
```

in /etc/pf.lb2.conf I have


```
lan_net = "10.0.1.0/24"
int_if  = "vr0"
ext_if1 = "ath0"
ext_if2 = "ath1"
ext_gw1 = "192.168.1.65"
ext_gw2 = "192.168.2.65"
#  nat outgoing connections on each internet interface
nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2)
#  default deny
#block in  from any to any
#block out from any to any
#  pass all outgoing packets on internal interface
pass out on $int_if from any to $lan_net
#  pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
#  load balance outgoing tcp traffic from internal network.
pass in on $int_if route-to \
  { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
  proto tcp from $lan_net to any flags S/SA modulate state
#  load balance outgoing udp and icmp traffic from internal network
pass in on $int_if route-to \
  { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
  proto { udp, icmp } from $lan_net to any keep state
#  general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state
#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any
```

and when I try and use the machine as a gateway I have no Internet connectivity.

After running `route change default 192.168.1.65` or `route change default 192.168.2.65`
`lynx http://whatismyip.org` works fine so I know its my pf.conf

Also pflog has nothing to say


```
X# tcpdump -r /var/log/pflog
reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file)
X#
```

I am now offering to pay someone to get this working.


----------



## DutchDaemon (Oct 21, 2009)

pflog has nothing to say if you don't put log statements in some of your rules ...


----------



## dextro_ (Oct 21, 2009)

ooooooooo

Well here is the pf conf I am now using


```
lan_net = "10.0.0.0/24"
int_if  = "vr0"
ext_if1 = "ath1"
ext_if2 = "ath0"
ext_gw1 = "192.168.2.65"
ext_gw2 = "192.168.1.65"

#  options
set block-policy return
set loginterface $int_if
set skip on lo

#  scrub
scrub in

#  nat outgoing connections on each internet interface
nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2)

#  default deny
#block in  from any to any
#block out from any to any

#  pass all outgoing packets on internal interface
pass out log on $int_if from any to $lan_net

#  pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if

#  load balance outgoing tcp traffic from internal network.
pass in log on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto tcp from $lan_net to any flags S/SA modulate state

#  load balance outgoing udp and icmp traffic from internal network
pass in log on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto { udp, icmp } from $lan_net to any keep state

#  general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any
```

Still doesn't work and here is pflog


```
X# tcpdump -n -e -ttt -r /var/log/pflog
reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file)
000000 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
62. 025430 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
179. 037496 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
64. 516376 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
52. 512273 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
68. 007190 rule 4294967295/8(ip-option): pass in on ath0: 192.168.1.64 > 239.255.255.250: igmp v2 report 239.255.255.250
45809. 095211 rule 3/0(match): pass in on vr0: 10.0.0.21.63371 > 4.2.2.1.53: 53331+[|domain]
000133 rule 3/0(match): pass in on vr0: 10.0.0.21.63371 > 4.2.2.2.53: 53331+[|domain]
4. 570419 rule 3/0(match): pass in on vr0: 10.0.0.21.64909 > 4.2.2.1.53: 18222+[|domain]
180250 rule 3/0(match): pass in on vr0: 10.0.0.21.53685 > 4.2.2.1.53: 62793+[|domain]
813257 rule 3/0(match): pass in on vr0: 10.0.0.21.64909 > 4.2.2.2.53: 18222+[|domain]
187690 rule 3/0(match): pass in on vr0: 10.0.0.21.53685 > 4.2.2.2.53: 62793+[|domain]
048105 rule 3/0(match): pass in on vr0: 10.0.0.2.138 > 10.0.0.255.138: NBT UDP PACKET(138)
1. 208148 rule 3/0(match): pass in on vr0: 10.0.0.21.54945 > 4.2.2.1.53: 12119+ A? wpad. (22)
994928 rule 3/0(match): pass in on vr0: 10.0.0.21.54945 > 4.2.2.2.53: 12119+ A? wpad. (22)
11. 642613 rule 3/0(match): pass in on vr0: 10.0.0.21.58059 > 4.2.2.1.53: 20495+[|domain]
131439 rule 3/0(match): pass in on vr0: 10.0.0.21.63315 > 4.2.2.1.53: 30012+[|domain]
862971 rule 3/0(match): pass in on vr0: 10.0.0.21.58059 > 4.2.2.2.53: 20495+[|domain]
124973 rule 3/0(match): pass in on vr0: 10.0.0.21.63315 > 4.2.2.2.53: 30012+[|domain]
1. 257489 rule 3/0(match): pass in on vr0: 10.0.0.21.137 > 10.0.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
2. 245281 rule 3/0(match): pass in on vr0: 10.0.0.21.56749 > 4.2.2.1.53: 19791+[|domain]
1. 000956 rule 3/0(match): pass in on vr0: 10.0.0.21.56749 > 4.2.2.2.53: 19791+[|domain]
9. 508820 rule 3/0(match): pass in on vr0: 10.0.0.21.53949 > 4.2.2.1.53: 57264+[|domain]
985000 rule 3/0(match): pass in on vr0: 10.0.0.21.53949 > 4.2.2.2.53: 57264+[|domain]
3. 507580 rule 3/0(match): pass in on vr0: 10.0.0.21.63316 > 4.2.2.1.53: 38313+ A? wpad. (22)
995959 rule 3/0(match): pass in on vr0: 10.0.0.21.63316 > 4.2.2.2.53: 38313+ A? wpad. (22)
9. 542193 rule 3/0(match): pass in on vr0: 10.0.0.21.50897 > 4.2.2.1.53: 55322+[|domain]
999664 rule 3/0(match): pass in on vr0: 10.0.0.21.50897 > 4.2.2.2.53: 55322+[|domain]
4. 558376 rule 3/0(match): pass in on vr0: 10.0.0.21.53680 > 4.2.2.1.53: 56167+[|domain]
270216 rule 3/0(match): pass in on vr0: 10.0.0.21.63972 > 4.2.2.1.53: 9717+[|domain]
719435 rule 3/0(match): pass in on vr0: 10.0.0.21.53680 > 4.2.2.2.53: 56167+[|domain]
159883 rule 3/0(match): pass in on vr0: 10.0.0.21.51951 > 4.2.2.1.53: 8189+[|domain]
127171 rule 3/0(match): pass in on vr0: 10.0.0.21.63972 > 4.2.2.2.53: 9717+[|domain]
874852 rule 3/0(match): pass in on vr0: 10.0.0.21.51951 > 4.2.2.2.53: 8189+[|domain]
12. 908784 rule 3/0(match): pass in on vr0: 10.0.0.21.50441 > 4.2.2.1.53: 3796+[|domain]
288720 rule 3/0(match): pass in on vr0: 10.0.0.21.63486 > 4.2.2.1.53: 38903+[|domain]
700832 rule 3/0(match): pass in on vr0: 10.0.0.21.50441 > 4.2.2.2.53: 3796+[|domain]
304220 rule 3/0(match): pass in on vr0: 10.0.0.21.63486 > 4.2.2.2.53: 38903+[|domain]
1. 226810 rule 3/0(match): pass in on vr0: 10.0.0.21.58514 > 4.2.2.1.53: 20819+ A? wpad. (22)
1. 054006 rule 3/0(match): pass in on vr0: 10.0.0.21.58514 > 4.2.2.2.53: 20819+ A? wpad. (22)
16. 231589 rule 3/0(match): pass in on vr0: 10.0.0.21.50901 > 4.2.2.1.53: 12708+[|domain]
993350 rule 3/0(match): pass in on vr0: 10.0.0.21.50901 > 4.2.2.2.53: 12708+[|domain]
16. 088346 rule 3/0(match): pass in on vr0: 10.0.0.21.49319 > 4.2.2.1.53: 37629+ A? wpad. (22)
988060 rule 3/0(match): pass in on vr0: 10.0.0.21.49319 > 4.2.2.2.53: 37629+ A? wpad. (22)
14. 000227 rule 3/0(match): pass in on vr0: 10.0.0.21.137 > 10.0.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
2. 251811 rule 3/0(match): pass in on vr0: 10.0.0.21.64983 > 4.2.2.1.53: 15671+[|domain]
996251 rule 3/0(match): pass in on vr0: 10.0.0.21.64983 > 4.2.2.2.53: 15671+[|domain]
16. 037259 rule 3/0(match): pass in on vr0: 10.0.0.21.51516 > 4.2.2.1.53: 9643+ A? wpad. (22)
992207 rule 3/0(match): pass in on vr0: 10.0.0.21.51516 > 4.2.2.2.53: 9643+ A? wpad. (22)
13. 998453 rule 3/0(match): pass in on vr0: 10.0.0.21.137 > 10.0.0.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
2. 252469 rule 3/0(match): pass in on vr0: 10.0.0.21.64322 > 4.2.2.1.53: 6987+[|domain]
996194 rule 3/0(match): pass in on vr0: 10.0.0.21.64322 > 4.2.2.2.53: 6987+[|domain]
X#
```


----------

