# PF rdr problem: please Help



## maraspin (Aug 10, 2009)

Hello everyone, I'm a new FreeBSD user and I'd like to activate port forwarding for 2 services (pop3 & https) from my A host (which handles other services on other ports), to some other hosts that I'll call B and C from now on, respectively.

The situation is similar to the following (please forgive my sloppy ASCII art):

```
...................../---\ .................
..Internet -->.......| G.| .................
.....................\___/......x.x.x.0 net.
...................... \....................
........................\...................                    
.........../---\......../---\........./---\. 
...........| C.|..<--...| A.|...-->...| B.|.
...........\___/...80...\___/...110...\___/.
............................................
..............\---  192.168.0.x net ---/....
```

So far all services were given out by host A; I am now switching to host B and C certain services, but for different reasons, I'm the position where I can not simply ask my users to change the IP for the server they're using.

So, I have the "gateway" in my draft, which is actually the host currently dishing off services with external IF bge0, IP xx.xx.xx.248 and internal IF lagg0, IP 192.168.41.48; host B and C are both Linux boxes and have IP addresses 192.168.41.50 and 192.168.41.40 respectively. All hosts are both on the 192.168.41.0 network (private) and on the xx.xx.xx.0 network, which exposes them to the internet through Gateway, which I'll call G.

I'd like to send requests directed to A on port 80 and 110 to hosts B and C while customers update their configurations for the 2 new IPs.

After reading some documentation online (in particular http://snipurl.com/pl80g and http://snipurl.com/pl81h), I have done the following:

1] enabled forwarding support in A, thought sysctl: 


```
belfast# sysctl -a | grep forward
kern.smp.forward_roundrobin_enabled: 1
kern.smp.forward_signal_enabled: 1
net.inet.ip.forwarding: 1
net.inet.ip.fastforwarding: 1
```

and put gateway_enable="YES" on my /etc/rc.conf file, so that 
forwarding support will be active after a reboot.

2] added the following rules to my pf.conf in A

NAT Rules Section:

```
rdr on $public_if proto tcp from any to xx.xx.xx.248 port 110 -> $popper port 110
rdr on $public_if proto tcp from any xx.xx.xx.248 port 80 -> $webserver port 80
```

FILTERING Section:

```
pass in on $public proto tcp from any to $public port 110
pass in on $public proto tcp from any to $public port 80
```

and then reloaded pf rules with pfctl -f.


The problem is that when I try to connect to A, on both ports 80 and 110, I get no valid responses; nmap from an external box moreover shows both ports in state filtered. 

If I do telnet/lynx... from my host A to B and C, I am able to reach and use the services they provide though; tcpdump show moreover traces of connections on such hosts when I try to connect from another box through the redirection, even though, I get no access to the services, at the end.


```
17:56:35.579510 IP REMOTEIP.m3ua > 192.168.41.54.http: S 385207113:385207113(0) win 65520 <mss 1260,nop,nop,sackOK>
17:56:41.574961 IP REMOTEIP.m3ua > 192.168.41.54.http: S 385207113:385207113(0) win 65520 <mss 1260,nop,nop,sackOK>
```


Routing is configured like this on my C host (similar configuration holds for B) and I suspect it might have something to do with my problem:


```
[root@dallas tmp]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.41.0   *  255.255.255.0  U     0      0        0 eth0
x.x.x.0        * 255.255.224.0   U     0      0        0 eth2
169.254.0.0    * 255.255.0.0     U     0      0        0 eth2
default      other-gateway 0.0.0.0   UG    0      0    0 eth2
```


I have spent the whole day documenting myself on pf, but I confess I am no network wizard and I'm sure I have left out some important detail.

I'd REALLY appreciate if some FreeBSD Guru could be so kind to give me some hint on this.


Thank you in advance,
Steve


----------



## DutchDaemon (Aug 10, 2009)

maraspin said:
			
		

> ```
> NAT Rules Section:
> [code]rdr on $public_if proto tcp from any to xx.xx.xx.248 port 110 -> $popper port 110
> rdr on $public_if proto tcp from any xx.xx.xx.248 port 80 -> $webserver port 80
> ...



NAT takes place before the pass rules, so use


```
rdr [B]pass[/B] on $public_if proto tcp from any to xx.xx.xx.248 port 110 -> $popper port 110
rdr [B]pass[/B] on $public_if proto tcp from any xx.xx.xx.248 port 80 -> $webserver port 80
pass in on $public proto tcp from any to [B]$popper[/B] port 110
pass in on $public proto tcp from any to [B]$webserver[/B] port 80
```

The 'rdr pass' takes care of allowing traffic to the public ports, the 'pass in' rules take care of allowing that traffic through to the back-end hosts. You may need additional 'pass out' rules for the next interface as well, if there is any.


----------



## maraspin (Aug 10, 2009)

Hello and thanks a lot for the lightning fast reply. Just tried what you've suggested me, but I still cannot reach my host, seeing it as filtered...

At this point, I suspect there might be some other problems involved; so, here it is my complete pf.conf, just in case...


```
#####################################################################
# Section 1 - MACRO Definitions

pubblica="bge0"
privata="lagg0"
webserver="192.168.41.54"
popper="192.168.41.50"
extaddress="x.x.x.248"

#####################################################################
# Section 2 - OPTIONS


#####################################################################
# Section 3 - SCRUB Rules

scrub in all


#####################################################################
# Section 4 - NAT Rules

rdr pass on $pubblica proto tcp from any to $extaddress port 110 -> $popper port 110
rdr pass on $pubblica proto tcp from any to $extaddress port 80 -> $webserver port 80

#####################################################################
# Section 5 - FILTERING Rules

# I block everything on public IF by default
block in on $pubblica
pass in on $privata

pass in on $pubblica proto tcp from any to $popper port 110
pass in on $pubblica proto tcp from any to $webserver port 80

# I allow SSH traffic on selected port
pass in on $pubblica proto tcp from any to $pubblica port 4222
```


Any suggestions? Thanks a lot, once again!

Steve


----------



## rturja (Aug 10, 2009)

maraspin said:
			
		

> ```
> pass in on $pubblica proto tcp from any to $popper port 110
> pass in on $pubblica proto tcp from any to $webserver port 80
> 
> ...


You should use

```
pass in on $pubblica proto tcp from any to $extaddress port 110
pass in on $pubblica proto tcp from any to $extaddress port 80
```

as on public interface the IP-address contacted is the external one, while the clients stay blissfully unaware on the IP's assigned to servers in the internal net


----------



## maraspin (Aug 10, 2009)

Thank you rturja for your reply; unfortunately I still can't do what I wish, as nmap on a remote host still tells me both of my ports are filtered...


----------



## rturja (Aug 10, 2009)

Couple more points - do you have nat rules in effect, you might want something like:


```
nat on $pubblica from 192.168.41.1/24 to any -> $pubblica
```

before the rdr's

Otherwise the firewall won't change the source IP to the firewall IP for the packets originating inside firewall and the clients see addresses that are nonroutable (and out of he reach!)over the standard internet.

You might consider adding state for the incoming connections too - and quick for stopping the pf rule evaluation at the rule like this:

```
pass in quick on $pubblica proto tcp from any to $extaddress port 110 keep state
pass in quick on $pubblica proto tcp from any to $extaddress port 80 keep state
```


----------



## maraspin (Aug 10, 2009)

Hello rturja, and once again thank you. I've tried this other suggestion you gave me, but unfortunately still no luck.

Something that I have noticed though is that your nat suggestion might be totally ignored (and I'm not sure on the consequences of this) in  my configuration.

In fact, now I have my A box (FreeBSD host called belfast) forwarding requests to C (boston) without natting (just the rdr -> I think this means that source address is NOT translated), so that replies from C should also be directly sent to the client, on the best route available... 

Well, I am doing some tests from home now (let's assume 1.1.1.1 is my home IP) and I'm getting the following, in boston (host C) through tcpdump usage:



```
[root@boston ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes21:32:45.075450 IP 1.1.1.1.ovtopmd > boston.http: S 3996303051:3996303051(0) win 65520 <mss 1260,nop,nop,sackOK>
21:32:47.951414 IP 1.1.1.1.ovtopmd > boston.http: S 3996303051:3996303051(0) win 65520 <mss 1260,nop,nop,sackOK>
21:32:53.941506 IP 1.1.1.1.ovtopmd > boston.http: S 3996303051:3996303051(0) win 65520 <mss 1260,nop,nop,sackOK>
```

where eth1 is my private net (192.168.41.0/24) interface.
Outbound traffic is then the following:


```
[root@boston ~]# tcpdump -i eth0 dst 1.1.1.1  and port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:32:45.075452 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:32:47.951461 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:32:49.038443 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:32:50.038498 IP boston.http > 1.1.1.1.qke-llc-v3: S 2204376025:2204376025(0) ack 3985870811 win 5840 <mss 1460,nop,nop,sackOK>
21:32:53.941554 IP boston.http > h1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:32:55.038831 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:33:07.038644 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:33:31.240294 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
21:33:38.240541 IP boston.http > 1.1.1.1.qke-llc-v3: S 2204376025:2204376025(0) ack 3985870811 win 5840 <mss 1460,nop,nop,sackOK>
21:34:19.442177 IP boston.http > 1.1.1.1.ovtopmd: S 2251184456:2251184456(0) ack 3996303052 win 5840 <mss 1460,nop,nop,sackOK>
```

it seems the host is doing the "right" thing, sending replies to the public interface, and therefore to the client directly, without any needing for NATting on the FreeBSD box then; or might this "right thing" be the source of my problem indeed? ...so that I'd better be Dnatting instead of port forwarding? 

I am very sorry for being not good at networking and making these questions; I thank you for your time helping me out, I appreciate A LOT!

Steve


----------



## rturja (Aug 10, 2009)

The problem with your source addresses, when used in public internet is that they are from the intranet reserved blocks. So when the client gets 192.168.x.x as another endpoint the address cannot be routed over internet. That's where NAT kicks in, it overwrites the original intranet IP with the external IP of your firewall, so the clients think they are talking to the firewall, when they are actually talking to the servers inside.

Just to be sure - when you have tested the connection from outside, have you used the 192.168.x.x addresses or the firewall external interface one?


----------



## maraspin (Aug 10, 2009)

You're definitely right, rturja; by using tcpdump with the -n option, I now have seen the following:


```
[root@boston ~]# tcpdump -n -i eth0 dst 1.1.1.1  and port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:59:50.996813 IP 192.168.41.41.http > 1.1.1.1.joltid: S 3953614594:3953614594(0) ack 150600427 win 5840 <mss 1460,nop,nop,sackOK>
21:59:53.995084 IP 192.168.41.41.http > 1.1.1.1.joltid: S 3953614594:3953614594(0) ack 150600427 win 5840 <mss 1460,nop,nop,sackOK>
21:59:54.052099 IP 192.168.41.41.http > 1.1.1.1.joltid: S 3953614594:3953614594(0) ack 150600427 win 5840 <mss 1460,nop,nop,sackOK>
```

...so that obviously I'm not getting my replies over the Internet; now I haven't got to the solution, but at least I've understood the problem... This takes me much closer to the solution now!!!

What was unknown to me is that the rdr rule is actually natting, so I guess I have now to put my hands on my Linux box and change source addresses if I want my packets to be delivered to my client across the Internet from there...

Or otherwise, just not to do anything on the Linux box (this is a temporary fix for me, so I'd really like to keep the modifications I make around as tight as possible), would it be possible to make a rdr to the external interface of my webserver (host C, boston)? I have read somewhere that it's not possible to do rdr on the same interface, having packets getting in and out... but I am definitely not an expert here, so that if someone can enlighten me on this, that'd be great.

If this could be possible, then I could just do:


```
rdr pass on $pubblica proto tcp from any to $extaddress port 110 -> $webserver_publicip_address port 80
```

and that'd do the job for me... But again, I've read something that stated that this approach wouldn't have worked, so I refrained so far...

If someone thinks this might be the way to go or has some other ideas to achieve what I'm trying to do, I'm VERY thankful in advance.


Thanks a lot to you all for reading and helping me out,
Steve


----------



## maraspin (Aug 10, 2009)

I have now tried what I've just thought of... The strange thing is that I see incoming connection requests to my webserver through tcpdump:


```
[root@boston ~]# tcpdump -n -i eth0 src 1.1.1.1 and dst x.x.x.241 and port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:16:44.938309 IP 1.1.1.1.taiclock > x.x.x.241.http: S 423541192:423541192(0) win 65520 <mss 1260,nop,nop,sackOK>
22:16:44.999064 IP 1.1.1.1.taiclock > x.x.x.241.http: R 423541193:423541193(0) win 0
22:16:47.893082 IP 1.1.1.1.taiclock > x.x.x.241.http: S 423541192:423541192(0) win 65520 <mss 1260,nop,nop,sackOK>
22:16:47.952808 IP 87.14.25.122.taiclock > 62.149.202.241.http: R 423541193:423541193(0) win 0
22:16:53.885081 IP 1.1.1.1.taiclock > x.x.x.241.http: S 423541192:423541192(0) win 65520 <mss 1260,nop,nop,sackOK>
22:16:53.943023 IP 1.1.1.1.taiclock > x.x.x.241.http: R 423541193:423541193(0) win 0
```

and there seem to be replies also:


```
[root@boston ~]# tcpdump -n -i eth0 dst 1.1.1.1  and port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:14:36.673377 IP x.x.x.241.http > 1.1.1.1.3976: S 605473416:605473416(0) ack 387894503 win 5840 <mss 1460,nop,nop,sackOK>
22:15:39.883642 IP x.x.x.241.http > 1.1.1.1.bv-smcsrv: S 668987869:668987869(0) ack 406406121 win 5840 <mss 1460,nop,nop,sackOK>
22:15:42.891436 IP x.x.x.241.http > 1.1.1.1.bv-smcsrv: S 671995664:671995664(0) ack 406406121 win 5840 <mss 1460,nop,nop,sackOK>
22:15:48.882969 IP x.x.x.241.http > 1.1.1.1.bv-smcsrv: S 677987194:677987194(0) ack 406406121 win 5840 <mss 1460,nop,nop,sackOK>
22:16:44.938421 IP x.x.x.241.http > 1.1.1.1.taiclock: S 719235034:719235034(0) ack 423541193 win 5840 <mss 1460,nop,nop,sackOK>
22:16:47.893131 IP x.x.x.241.http > 1.1.1.1.taiclock: S 722189763:722189763(0) ack 423541193 win 5840 <mss 1460,nop,nop,sackOK>
22:16:53.885146 IP x.x.x.241.http > 1.1.1.1.taiclock: S 728181776:728181776(0) ack 423541193 win 5840 <mss 1460,nop,nop,sackOK>
```

...but still I can't see the web page... 

I'm thinking though that if it was merely a PF problem, I shouldn't be seeing requests coming and therefore I suspect there's something else in the picture I'm not getting... 

As always, any help would be GREATLY appreciated!


----------



## DutchDaemon (Aug 10, 2009)

Try adding 'set skip on lo0'.


----------



## DutchDaemon (Aug 10, 2009)

And *please* use 
	
	



```
tags around system output.
```


----------



## maraspin (Aug 10, 2009)

Tnx DD, trying out right now, even if I'm afraid that the problem might be trickier. But I'll post my results here anyway.

Sorry about the 
	
	



```
tags, I was actually looking for a way to use them... now I hope know how :-)
```


----------



## maraspin (Aug 11, 2009)

Hello everyone, I think I have found where my problem is but unfortunately I also think I still need a hand to solve it.

The fact is that using rdr address translation does not seem to happen and therefore the connections getting to my C host come from a public Internet address, to which host C directly tries to reply. The problem here is that I have gateway G through which packets to the Internet are directly sent and therefore a socket can not be not established (on host A -> remote client).

At this point, my idea would be that of translating requests to A on port 80 and 110 to a private source address (that of A) to be then forwarded to B and C so that their replies would get back directly to A and then it could make a socket with the remote client.

Therefore I have tried to do:


```
pubblica="bge0"
webserver="192.168.41.41"
extaddress="x.x.x.248"

[B]nat on $pubblica proto tcp from any to $extaddress port 80 -> $webserver port 80[/B]
```

but unfortunately I see no traffic coming on the webserver internal interface using tcpdump (where I expect my packets forwarded by A to come).

This is what my FreeBSD box shows using pfctl:


```
belfast# pfctl -s state
No ALTQ support in kernel
ALTQ related functions disabled
all udp 255.255.255.255:67 <- 0.0.0.0:68       NO_TRAFFIC:SINGLE
all tcp x.x.x.248:4222 <- 1.1.1.1:1935       ESTABLISHED:ESTABLISHED
all tcp x.x.x.248:80 <- 1.1.1.1:3903       TIME_WAIT:TIME_WAIT
```

I was hoping that x.x.x.248:80 <- 1.1.1.1:3903 would be directed to my webserver at its internal interface, but this does not happen.

Any help on what I consider DNATting at this point would be GREATLY appreciated!

You all have a nice day


----------

