# Forward HTTP traffic to internal web servers



## samsam9988 (Sep 11, 2014)

Hi, I own two public address x.x.x.x and y.y.y.y for my two web servers. Another z.z.z.z public address is assigned to tun0. A FreeBSD 10 system is configured as a gateway to forward traffic from tun0 to the two internet web servers. Illustrated as below:

```
x.x.x.x
z.z.z.z -- tun0 ---- 2 web servers (192.168.5.20 via em2 and 192.168.6.20 via em3)
y.y.y.y
```
However when I browsed x.x.x.x from my iPhone (outside my local network), traffic is not getting through to the web servers. Same as when I browsed y.y.y.y.I am wondering what should I do to make this happen?

My rc.conf configuraiton:

```
gateway_enable="YES"

inetd_enable="YES"
kern_securelevel_enable="NO"

hostname="iExtentGateway"
#ifconfig_em0="DHCP"
#ifconfig_em4="inet 192.168.1.254 netmask 255.255.255.0"
#ifconfig_em4="inet 10.0.1.254 netmask 255.255.255.0"
ifconfig_em0="inet 192.168.2.254 netmask 255.255.255.0"
ifconfig_em1="inet 192.168.1.254 netmask 255.255.255.0"
ifconfig_em2="inet 192.168.5.254 netmask 255.255.255.0"
ifconfig_em3="inet 192.168.6.254 netmask 255.255.255.0"

ppp_enable="yes"
ppp_profile="pppoe"
ppp_mode="ddial"


natd_enable="YES"
natd_flags="-f /etc/natd.conf"
firewall_enable="YES"
firewall_logging="YES"
firewall_script="/etc/ipfw.rules"
```

natd.conf file:

```
interface tun0
use_sockets yes
dynamic yes
redirect_address 192.168.5.20 x.x.x.x
redirect_address 192.168.6.20 y.y.y.y
```

ipfw.rules file:

```
#!/bin/sh

IPF="ipfw -q add"
ipfw -q -f flush
skip="skipto 800"

$IPF divert natd all from any to any via tun0

#loopback
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# statefull
$IPF 50 check-state
$IPF 60 allow tcp from any to any established
$IPF 70 allow all from any to any out keep-state
$IPF 80 allow icmp from any to any
# open port ftp (20,21), ssh (22), mail (25)
# http (80), dns (53) etc
$IPF 130 allow tcp from any to any 22 in
$IPF 140 allow tcp from any to any 22 out
$IPF 150 allow tcp from any to any 25 in
$IPF 160 allow tcp from any to any 25 out
$IPF 170 allow udp from any to any 53 in
$IPF 175 allow tcp from any to any 53 in
$IPF 180 allow udp from any to any 53 out
$IPF 185 allow tcp from any to any 53 out
$IPF 200 allow tcp from any to any 80 in
$IPF 210 allow tcp from any to any 80 out
$IPF 720 allow all from 192.168.0.0/16 to any keep-state
$IPF 722 allow all from 10.0.0.0/8 to any keep-state
$IPF 730 allow tcp from any to x.x.x.x 80 keep-state
$IPF 740 allow tcp from any to y.y.y.y 80 keep-state

# deny and log everything
$IPF 900 deny log all from any to any
```
Very appreciate suggestion and help.
Best regards
Sam


----------



## SirDice (Sep 11, 2014)

samsam9988 said:
			
		

> However when I browsed x.x.x.x from my iphone (outside my local network), traffic is not getting through to the web servers. Same as when I browsed y.y.y.y


Is this traffic actually routed to your FreeBSD host? Does it even arrive there?


----------



## samsam9988 (Sep 11, 2014)

SirDice said:
			
		

> samsam9988 said:
> 
> 
> 
> ...


Yes it is arriving tun0:


```
root@iExtentGateway:/etc # tcpdump -i tun0 dst y.y.y.y
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type NULL (BSD loopback), capture size 65535 bytes
01:16:48.781904 IP a12-09-03.opera-mini.net.48780 > y.y.y.y.static.exetel.com.au.http: Flags [S], seq 3633602986, win 14600, options [mss 1440,sackOK,TS val 2444293098 ecr 0,nop,wscale 7], length 0

^C
4 packets captured
2315 packets received by filter
0 packets dropped by kernel
root@iExtentGateway:/etc # tcpdump -i tun0 dst x.x.x.x
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type NULL (BSD loopback), capture size 65535 bytes
01:17:14.332614 IP a12-09-03.opera-mini.net.52264 > x.x.x.x.static.exetel.com.au.http: Flags [S], seq 4109861018, win 14600, options [mss 1440,sackOK,TS val 2444299486 ecr 0,nop,wscale 7], length 0
```


----------



## samsam9988 (Sep 12, 2014)

samsam9988 said:
			
		

> SirDice said:
> 
> 
> 
> ...



I tried to add the following rules in /etc/ipfw.rules:

```
00200 divert natd ip from x.x.x.x 80 to 192.168.5.20 dst-port 80 via tun0
00300 divert natd ip from y.y.y.y 80 to 192.168.6.20 dst-port 80 via tun0
```
or 

```
00200 divert natd ip from x.x.x.x 80 to 192.168.5.20 dst-port 80 
00300 divert natd ip from y.y.y.y 80 to 192.168.6.20 dst-port 80
```
But incoming HTTP traffic to x.x.x.x and y.y.y.y is not being forwarded to my internal servers.


----------



## SirDice (Sep 12, 2014)

I don't use IPFW but on PF the redirections are done like this:

```
rdr on $ext_if inet proto tcp from any to $addr1 port 80 -> 192.168.5.20 port 80
rdr on $ext_if inet proto tcp from any to $addr2 port 80 -> 192.168.6.20 port 80
```

Which roughly translates as "traffic from anywhere to IP $addr2 port 80 coming in on $ext_if needs to be redirected to 192.168.5.20 port 80".


----------



## samsam9988 (Sep 12, 2014)

SirDice said:
			
		

> I don't use IPFW but on PF the redirections are done like this:
> 
> ```
> rdr on $ext_if inet proto tcp from any to $addr1 port 80 -> 192.168.5.20 port 80
> ...



PF in FreeBSD 10 is deadly slow. It becomes useless.
that s I switched back to IPFW.


----------



## SirDice (Sep 12, 2014)

samsam9988 said:
			
		

> PF in FreeBSD 10 is deadly slow. It becomes useless.


I doubt it's slow as FreeBSD 10 had a major update on PF to be able to use SMP more efficiently. But there could of course be bugs. Have you tried 10-STABLE or the 10.1-PRERELEASE?


----------



## kpa (Sep 12, 2014)

samsam9988 said:
			
		

> SirDice said:
> 
> 
> 
> ...



Care to share your setup and explain your testing methods for the "slowness". This is about the first time I've seen anyone claim PF on FreeBSD 10 as slow.


----------



## samsam9988 (Sep 12, 2014)

kpa said:
			
		

> samsam9988 said:
> 
> 
> 
> ...



*S*orry in FreeBSD 9.2-RELEASE.


----------



## samsam9988 (Sep 12, 2014)

> > > PF in FreeBSD 10 is deadly slow. It becomes useless.
> > > that s I switched back to IPFW.
> >
> >
> ...



The setup in FreeBSD 9.2 release is as shown below:

```
root@iExtentGateway:/etc # cat pf.conf
#       $FreeBSD: release/9.2.0/share/examples/pf/pf.conf 218854 2011-02-19 14:57:00Z brucec $
#       $OpenBSD: pf.conf,v 1.34 2007/02/24 19:30:59 millert Exp $
#
# See pf.conf(5) and /usr/share/examples/pf for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.

tcp_services = "{http, ssh, imaps, smtp, domain, ntp, www, https, 18080, 1444}"
udp_services= "{domain, ntp}"

#ext_if="em4"
ext_if="tun0"
int_if="em0"
webservice_if="em1"
seacell_ip="x.x.x.x"
morfeus_ip="y.y.y.y"
iextserv_ip="192.168.1.20"
gateway_ip="z.z.z.z"
seacell_if="em2"
morfeus_if="em3"
iextserv_if="em1"

#table <spamd-white> persist

set skip on lo

scrub in

nat on $ext_if from !($ext_if) -> ($ext_if:0)

rdr pass on $ext_if proto tcp from any to $seacell_ip port 18080 -> 192.168.1.20 port 18080
rdr pass on $ext_if proto tcp from any to $morfeus_ip port 18080 -> 192.168.1.20 port 18080
# for jboss webservice runtime.... keep this redirection even for go live...
rdr pass on $ext_if proto tcp from any to $seacell_ip port 14447 -> 192.168.1.20 port 14447
rdr pass on $ext_if proto tcp from any to $morfeus_ip port 14447 -> 192.168.1.20 port 14447

rdr pass on $ext_if proto tcp from any to $seacell_ip port 80 -> 192.168.5.20 port 80
rdr pass on $ext_if proto tcp from any to $morfeus_ip port 80 -> 192.168.6.20 port 80
rdr pass on $ext_if proto tcp from any to $seacell_ip port 22 -> 192.168.5.20 port 22 
rdr pass on $ext_if proto tcp from any to $morfeus_ip port 22 -> 192.168.6.20 port 22
rdr pass on $ext_if proto tcp from any to $seacell_ip port 53 -> 192.168.5.20 port 53 
rdr pass on $ext_if proto tcp from any to $morfeus_ip port 53 -> 192.168.6.20 port 53 
rdr pass on $ext_if proto udp from any to $seacell_ip port 53 -> 192.168.5.20 port 53 
rdr pass on $ext_if proto udp from any to $morfeus_ip port 53 -> 192.168.6.20 port 53 
rdr pass on $ext_if proto icmp from any to $seacell_ip -> 192.168.5.20 
rdr pass on $ext_if proto icmp from any to $morfeus_ip -> 192.168.6.20

antispoof quick for $ext_if

#anchor "ftp-proxy/*"
block in log
pass out quick modulate state

#pass everything for internal server and pcs:
pass quick on $webservice_if modulate state
pass quick on $int_if modulate state

#antispoof quick for { lo $int_if }

#pass incoming www trafic:
pass quick on $ext_if proto tcp from any to any port $tcp_services modulate state
pass quick on $ext_if proto udp from any to any port $udp_services modulate state


pass quick on $seacell_if from any to any modulate state
pass quick on $morfeus_if from any to any modulate state
pass quick on $iextserv_if from any to any modulate state

#pass incoming ssh:
pass in quick on $ext_if proto tcp to ($ext_if) port ssh modulate state

#pass in log on $ext_if proto tcp to ($ext_if) port smtp

#pass out smtp:
pass out log on $ext_if proto tcp from ($ext_if) to port smtp
pass out log on $seacell_if proto tcp from ($seacell_if) to port smtp
pass out log on $morfeus_if proto tcp from ($morfeus_if) to port smtp

#pass incoimg pings:
pass in on $ext_if inet proto icmp from any to ($ext_if) icmp-type { unreach, redir, timex }
pass in on $seacell_if inet proto icmp from any to ($seacell_if) icmp-type { unreach, redir, timex }
pass in on $morfeus_if inet proto icmp from any to ($morfeus_if) icmp-type { unreach, redir, timex }
```
The slowness is happen gradually.


----------



## SirDice (Sep 12, 2014)

What kind of bandwidth usage and how many concurrent connections are there? If you have a lot of connections you may simply be running out of memory for the state-tables. The slowness may also be caused by the VPN. What are you using for VPN?


----------



## samsam9988 (Sep 12, 2014)

SirDice said:
			
		

> What kind of bandwidth usage and how many concurrent connections are there? If you have a lot of connections you may simply be running out of memory for the state-tables. The slowness may also be caused by the VPN. What are you using for VPN?



Nah, nothing to do with my enriovvironment. I am running IPFW, it is super fast.


----------



## samsam9988 (Sep 12, 2014)

PF maybe can't handle "so-many" network interfaces.


----------



## samsam9988 (Sep 12, 2014)

samsam9988 said:
			
		

> PF may be can't handle "so-many" network interfaces.


Can anyone suggest a working example for redirecting incoming HTTP traffic to an internal web server?

I tried to add the following rule in IPFW:


```
$CMD divert natd all from any to x.x.x.x 80 dst 192.168.5.20 80
```
or

```
$CMD divert natd all from an to x.x.x.x 80 redivert-to 192.168.1.5.20 80
```

This is only a sudopseudo-code rather than a working solution. I don't know what the exact IPFW syntax for doing that.

Best regards
Sam


----------



## nforced (Sep 13, 2014)

samsam9988 said:
			
		

> PF in FreeBSD 9.2 is deadly slow. It becomes useless.
> that s I switched back to IPFW.



I don't remember having such problem with 9.2 and I am now on 10 and I can say it's deadly fast. 
What kind of a hardware do you have? I use http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt-dp.html and I reach speeds up to 950Mbits on my modest machine.


----------



## samsam9988 (Sep 13, 2014)

nforced said:
			
		

> samsam9988 said:
> 
> 
> 
> ...



How many LAN ports do you use?


----------



## obsigna (Sep 13, 2014)

samsam9988 said:
			
		

> ...
> natd.conf file:
> 
> ```
> ...



Reading the manual of natd(8), it seems to me that the interface directive in your natd.conf does not exactly what you intend it to do, namely natting all traffic that passes by tun0.



			
				http://www.freebsd.org/cgi/man.cgi?query=natd&sektion=8&n=1 said:
			
		

> -interface	| -n interface
> Use interface to determine the	aliasing address.  If there is
> a possibility that the	IP address associated with interface
> may change, the -dynamic option should	also be	used.  If this
> ...



Above means to me clearly, that in your setup, only the traffic from/to z.z.z.z is natted, which is not what you want.

I am not able to test the following, since I got only one public IP, and beside that, I use the NAT that is builtin to ipfw(8). Anyway, at  least it should give you an idea:

```
use_sockets yes
dynamic yes

# default instance
# public1
port 8668
alias_address x.x.x.x
redirect_address 192.168.5.20 x.x.x.x

# second instance
instance public2
port 8888
alias_address y.y.y.y
redirect_address 192.168.6.20 y.y.y.y
```


----------



## nforced (Sep 13, 2014)

samsam9988 said:
			
		

> How many LAN ports do you use?


This is WAN -> LAN both ports on one card if this is what you ask.


----------



## samsam9988 (Sep 14, 2014)

nforced said:
			
		

> samsam9988 said:
> 
> 
> 
> ...


I have 5


----------



## nforced (Sep 14, 2014)

samsam9988 said:
			
		

> I have 5


So you are saying if I have 5 the performance is going to be unacceptable? 
You didn't said what kind of hardware do you have and why won't you try FreeBSD 10 as pf have major optimization there?
Maybe someone who have 5 or more ports can share his experience with pf, I am curious about this


----------



## samsam9988 (Sep 14, 2014)

nforced said:
			
		

> samsam9988 said:
> 
> 
> 
> ...




```
# ifconfig -akk
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:15:17:f2:e5:91
        inet 192.168.2.254 netmask 0xffffff00 broadcast 192.168.2.255
        inet6 fe80::215:17ff:fef2:e591%em0 prefixlen 64 scopeid 0x1 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:f2:e5:90
        inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::215:17ff:fef2:e590%em1 prefixlen 64 scopeid 0x2 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:f2:e5:93
        inet 192.168.5.254 netmask 0xffffff00 broadcast 192.168.5.255
        inet6 fe80::215:17ff:fef2:e593%em2 prefixlen 64 scopeid 0x3 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:15:17:f2:e5:92
        inet 192.168.6.254 netmask 0xffffff00 broadcast 192.168.6.255
        inet6 fe80::215:17ff:fef2:e592%em3 prefixlen 64 scopeid 0x4 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 44:87:fc:73:c8:f7
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
```


----------

