# Multiple WAN interfaces for incoming traffic



## oobayly (Aug 11, 2009)

Currently using FreeBSD 7.2-Release with all ports up to date as my firewall.

I've got a firewall/router with 3 NICs: 2x ADSL & 1 LAN. So far I've only been using one of the lines at a time, with the 2nd as a backup.
Both ADSL lines have static IPs so I thought it's be handly to have the DNS record for our office to provide both IPs, this way even if one connection is down, the office can still be accessed remotely.

My current pf.conf works fine, but for only a single connection, so I used an example pf.conf to configure a round-robin load balanced setup. Outbound traffic works perfectly, with the traffic using both ADSL lines.

However I've had some issues with setting up the incoming traffic. In short, an incoming connection works fine if it's to the ADSL line that is the default gateway (tun1). If I attempt to access the office via the secondary ADSL line (tun0), I see the traffic coming in, but it is sent back out of the office on the primary ADSL line (tun1).

How come traffic is being sent out the default gateway rather than the interface it came in on? What am I missing in (this very simple) ruleset? 

*The ISP for tun1 is actually routing the traffic back to the originator even though the source isn't correct, which in my opinion they shouldn't do. It works but obviously it can't be relied on. If however I swap the default gateway and rerun the test, the ISP tun0 doesn't route incorrectly sourced packets.


_ifconfig_

```
tun0: flags=8151<UP,POINTOPOINT,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1492
        inet xx.xx.115.90 --> xx.xx.112.1 netmask 0xffffffff
        Opened by PID 462
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        inet xx.xx.82.202 --> xx.xx.111.216 netmask 0xffffffff
        Opened by PID 470
```


_netstat -rnf inet | grep default_

```
default            xx.xx.111.216     UGS         0    33943   tun1
```


Execute command from remote server (xx.xx.33.85)
_telnet xx.xx.82.202 22_

```
# tcpdump -ni tun0 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type NULL (BSD loopback), capture size 96 bytes

# tcpdump -ni tun1 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type NULL (BSD loopback), capture size 96 bytes
19:00:21.138403 IP xx.xx.33.85.2966 > xx.xx.82.202.22: S 1783025215:1783025215(0) win 65535 <mss 1380,nop,nop,sackOK>
19:00:21.138806 IP xx.xx.82.202.22 > xx.xx.33.85.2966: S 1046433002:1046433002(0) ack 1783025216 win 5840 <mss 1460,nop,nop,sackOK>
19:00:21.404552 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 1 win 65535
19:00:21.405412 IP xx.xx.82.202.22 > xx.xx.33.85.2966: P 1:26(25) ack 1 win 5840
19:00:21.728913 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 26 win 65510
19:00:24.253779 IP xx.xx.33.85.2966 > xx.xx.82.202.22: F 1:1(0) ack 26 win 65510
19:00:24.254186 IP xx.xx.82.202.22 > xx.xx.33.85.2966: F 26:26(0) ack 2 win 5840
19:00:24.424030 IP xx.xx.33.85.2966 > xx.xx.82.202.22: . ack 27 win 65510
```


Execute command from remote server (xx.xx.33.85)
_telnet xx.xx.115.90 22_

```
# tcpdump -ni tun0 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type NULL (BSD loopback), capture size 96 bytes
19:01:08.721838 IP xx.xx.33.85.2967 > xx.xx.115.90.22: S 2245618778:2245618778(0) win 65535 <mss 1380,nop,nop,sackOK>
19:01:08.779974 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 3546884040 win 65535
19:01:09.052440 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 41 win 65495
19:01:10.676351 IP xx.xx.33.85.2967 > xx.xx.115.90.22: F 0:0(0) ack 41 win 65495
19:01:10.736404 IP xx.xx.33.85.2967 > xx.xx.115.90.22: . ack 42 win 65495

# tcpdump -ni tun1 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type NULL (BSD loopback), capture size 96 bytes
19:01:08.722172 IP xx.xx.115.90.22 > xx.xx.33.85.2967: S 3546884039:3546884039(0) ack 2245618779 win 65535 <mss 1380,sackOK,eol>
19:01:08.820526 IP xx.xx.115.90.22 > xx.xx.33.85.2967: P 1:41(40) ack 1 win 65535
19:01:10.676488 IP xx.xx.115.90.22 > xx.xx.33.85.2967: . ack 2 win 65535
19:01:10.677654 IP xx.xx.115.90.22 > xx.xx.33.85.2967: F 41:41(0) ack 2 win 65535
```



pf.conf
Take from http://www.openbsd.org/faq/pf/pools.html, but modified for my interfaces & additional rules for ssh

```
lan_net = "xx.xx.0.0/22"
int_if  = "rl0"
ext_if1 = "tun0"
ext_if2 = "tun1"
ext_gw1 = "xx.xx.112.1"
ext_gw2 = "xx.xx.111.216"

#  nat outgoing connections on each internet interface
nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2)

rdr on $ext_if1 proto tcp from any to $ext_if1 port ssh -> localhost
rdr on $ext_if2 proto tcp from any to $ext_if2 port ssh -> localhost

#  default deny
block in  from any to any
block out from any to any

#  pass all outgoing packets on internal interface
pass out on $int_if from any to $lan_net
#  pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
#  load balance outgoing tcp traffic from internal network.
pass in log on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto tcp from $lan_net to any flags S/SA modulate state
#  load balance outgoing udp and icmp traffic from internal network
pass in log on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto { udp, icmp } from $lan_net to any keep state

#  general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

pass in on $ext_if1 proto tcp from any to any port ssh
pass in on $ext_if2 proto tcp from any to any port ssh

#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
pass out log on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any
pass out log on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any
```


----------



## DutchDaemon (Aug 11, 2009)

You need reply-to statements (pf.conf(5)) for your 'pass in' rules to enforce symmetric routing on incoming connections. You'll probably also want an if-bound state policy.

See http://kerneltrap.org/mailarchive/openbsd-misc/2007/7/14/152333 for an example.
(not entirely sure about the if-bound/floating weirdness in there, but try if-bound first anyway)


----------



## oobayly (Aug 12, 2009)

Brilliant, it's a pity the OpenBSD pages didn't give an example of that. I've now got everything working perfectly as well as the ftp-proxy issue you helped me with earlier.

Now I just have to have a good look through the man pages & docs to understand what exactly the if-bound policy does.

For the moment, I've had to keep the gateway ips in tables, so that I can edit them once the rule-set has been loaded. This is due to my ISPs providing a different gateway whenever ppp has to redial. I'm currently using ppp.linkup to update the tables. This is the only method I found suggested when dealing with dynamic gateways.

Thanks for all your help.


----------



## DutchDaemon (Aug 12, 2009)

oobayly said:
			
		

> Now I just have to have a good look through the man pages & docs to understand what exactly the if-bound policy does.



Stated simply: it will only create state for a connection on one interface at a time: on the interface the connection 'came in on'. This will prevent traffic from 'leaking out' of the wrong interface. 

The downside is that, instead of having the packet delivered (albeit using asymmetric routing using 'the wrong interface'), the packet won't be able to get out (the other interface doesn't have a state for that connection, so it refuses to pass the packet).

The 'floating' policy allows states for connections on the other interfaces, so the packet will be allowed out on any interface.

(switch in/out for traffic initiated locally)


----------



## SirDice (Aug 12, 2009)

oobayly said:
			
		

> Both ADSL lines have static IPs so I thought it's be handly to have the DNS record for our office to provide both IPs, this way even if one connection is down, the office can still be accessed remotely.


Keep in mind that DNS usually works in a round-robin way. Meaning if one of your connections goes down you'll get connection problems 50% of the times you try to connect.


----------



## oobayly (Aug 12, 2009)

SirDice said:
			
		

> Keep in mind that DNS usually works in a round-robin way. Meaning if one of your connections goes down you'll get connection problems 50% of the times you try to connect.



This is something I looked at. By taking down one of the connections and attempting to connect (using several browsers, .Net tcp client, ssh & telnet) and monitoring the traffic using wireshark, I found that the client would attempt to connect to one of the IPs, and if that timed out it would move on to the next IP returned by the DNS lookup. As you said, it works in a round-robin way.
If one of the connections does go down, it will mean than connect times are greater as it has to wait for the attempt to timeout, but I've found that the client always manages to connect as long as at least one connection is up.


----------



## overmind (Sep 5, 2009)

You could make a script and from time to time ping the gatway of both lines. If you have no reply from one gateway, pass all traffic to the other line.


----------



## vivek (Sep 5, 2009)

I use pfSense (FreeBSD+PF combo) for failover and loadbalncing. It has a web-based front end. Few clicks and you are online. It takes care of everything.


----------



## frankpeng (Apr 24, 2012)

I tried pf*S*ense. It works with multiple WANs. But I cannot add php modules to run Drupal. I added 
	
	



```
CUSTOM_PKG_LIST=........
```
 and made many CD's. No pkg's are added into the CD's. Now I was trying *O*penBSD. I even could not build its kernel. Now I am coming back to FreeBSD to see if possible to make freebsd FreeBSD to do pf*S*ense and *O*penBSD jobs.


----------

