# FreeBSD 8.3 PF CARP redirect



## triumdh (May 23, 2012)

Hello,

I have a basic testing setup with four machines.


```
-[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
            -                                -
              -                            -
                 -[FreeBSD 8.3 gateway 2]-
```

I am testing CARP and port redirection. Gateway 1 and 2 have CARP interfaces.

gateway 1 rc.conf

```
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass PASSWORD 10.0.0.18 netmask 255.255.255.0"
ifconfig_igb0_name="wan0"        # ISP
ifconfig_wan0="10.0.0.17/24"
```

gateway 2 rc.conf

```
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass PASSWORD advskew 100 10.0.0.18 netmask 255.255.255.0"
ifconfig_igb0_name="wan0"        # ISP
ifconfig_wan0="10.0.0.16/24"
```

gateway 1 and 2 share same pf.conf

```
## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to wan0 port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state
```

When I set server 1 with the gateway 10.0.0.18 (the CARP interface for gateway 1) I cannot ssh on port 8822 to server 1 from the windows PC.

When I set server 1 with the gateway 10.0.0.17 (the actual interface interface for gateway 1) I can ssh on port 8822 to server 1 from the windows PC.

This seems to say that the CARP interface is the issue.

I have read that FreeBSD does not have the ability to bind a CARP interface to a specific interface. Is this the problem?

*ifconfig* on the gateways shows an apparent bind to the loopback address.


```
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 10.0.0.18 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 0
```

Any help always appreciated.


----------



## kisscool-fr (May 23, 2012)

Hi,

The CARP interface is not the issue, the issue is in your pf.conf.

Try something like that and tell us if it works.


```
carp_if="10.0.0.18"

## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to $carp_if port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state
```


----------



## triumdh (May 23, 2012)

I tried that with the same results. The connection works when I don't use the CARP IP as the default gateway on the server 1.

BTW What was your theory for adding that command to the pf.conf?


----------



## gkontos (May 24, 2012)

A simple tcpdump should tell you exactly what is wrong:

[CMD=""]# tcpdump -n -e -ttt -i pflog0[/CMD]

However, I believe that you need to do the NAT on the carp interface. You can try:


```
rdr on $carp_if inet proto { tcp, udp } from any to $carp_if port = 8822 -> 192.168.1.97 port 8822
```

See if this works.


----------



## kisscool-fr (May 24, 2012)

triumdh said:
			
		

> I tried that with the same results. The connection works when I don't use the CARP IP as the default gateway on the server 1.
> 
> *BTW What was your theory for adding that command to the pf.conf?*




Carp interfaces are just virtual interfaces, it's like an alias. If you want to filter the traffic by interfaces on pf, you have to do it on the physical interface that the carp interface depends. It's why in your pf.conf you write on PHYSICAL_INTERFACE.

On the other side if you want to allow or deny traffic by ip, you use the carp address. Take a look at this http://www.openbsd.org/faq/pf/carp.html#RulesetTips

And what I wrote before could be simplified like this:


```
## TRANSLATION RULES (NAT)
rdr on wan0 inet proto { tcp, udp } from any to carp0 port = 8822 -> 192.168.1.97 port 8822

## FILTER RULES 
pass out log all keep state
pass in log all keep state
```

Could you post the output of pfctl -vnf /etc/pf.conf with these lines ?


----------



## triumdh (May 24, 2012)

Thank you for the reply,

Result of [cmd=]pfctl -vnf /etc/pf.conf[/cmd]


```
SYN_ONLY = "S/FSRA"
set loginterface lan0
carp_if = "192.168.1.98"
rdr on wan0 inet proto tcp from any to 192.168.10.100 port = 8822 -> 10.0.0.97 port 8822
rdr on wan0 inet proto udp from any to 192.168.10.100 port = 8822 -> 10.0.0.97 port 8822
pass out log all flags S/SA keep state
pass in log all flags S/SA keep state
```

The result of [cmd=]tcpdump -n -e -ttt -i pflog0[/cmd] shows no traffic going to the destination server. All results of tcpdump on any interface show me that the traffic stops after going to the gateway. When I change the default gateway on the destination server to the physical interface on the gateway, tcpdump shows traffic going to the destination server.

The gateway CARP interface is on the server 1 (destination) server side of the gateway so


```
rdr on wan0 inet proto { tcp, udp } from any to carp0 port = 8822 -> 192.168.1.97 port 8822
```

wouldn't help. Sorry  wasn't clearer on the setup.



```
Wan0                   Carp0/Lan0
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
            -                                -
              -                            -
                 -[FreeBSD 8.3 gateway 2]-
               Wan0                   Carp0/Lan0
```

When I set the default gateway on "server 1" to the carp0 IP address SSH does not connect.
When I set the default gateway on "server 1" to the Lan0 IP address SSH does connect.


----------



## kisscool-fr (May 24, 2012)

It's very strange.

Where does these addresses come from? 192.168.10.100 and 10.0.0.97? 

Can you post a more detailed scheme with interfaces, addresses and netmasks of the client, the gateways and the server please?


----------



## gkontos (May 24, 2012)

Does your rc.conf include:


```
gateway_enable=YES"
```


----------



## kisscool-fr (May 25, 2012)

gkontos said:
			
		

> Does your rc.conf include:
> 
> 
> ```
> ...





> When I set server 1 with the gateway 10.0.0.17 (the actual interface interface for gateway 1) I can ssh on port 8822 to server 1 from the windows PC.




From what he said IP forwarding is enabled otherwise what he told would not work.


----------



## triumdh (May 25, 2012)

I think I gave some misinformation about one of the ip addresses. I also took out gateway 2. Just to clarify. 


```
wan0                   carp0/lan0
                 -[FreeBSD 8.3 gateway 1]-
               -                           -
             -                               -
[Windows PC]                                   [FreeBSD 8.3 server 1]
```


gateway 1


```
#ifconfig
lan0:   inet 192.168.1.251 netmask 0xffffff00 broadcast 192.168.1.255

wan0:   inet 10.0.0.17 netmask 0xffffff00 broadcast 10.0.0.255

carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 10.0.0.18 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 0
```

When I set the default gateway on server 1 to the carp0 interface, the PC cannot ssh to server 1.
When I set the default gateway on server 1 to the lan0 interface, the PC can ssh to server 1.

This rules out everything but CARP or PF. There is a minimal PF config allowing everything through.


----------



## plamaiziere (May 28, 2012)

triumdh said:
			
		

> I think I gave some misinformation about one of the ip addresses. I also took out gateway 2. Just to clarify.
> 
> 
> ```
> ...



You need two CARP interfaces, one on wan0 and one on lan0. You need also a pfsync interface to synchronize packet filter states between the two firewalls.

*S*omething like 


```
carp0 : 192.168.1.254/24 on lan0
carp1 : 10.0.0.254/24 on wan0
```

*S*et the sysctl net.inet.carp.preempt to '1' (see *man carp*)

Then use the CARP addresses for gateway.

Regards.


----------



## triumdh (Jun 5, 2012)

Thank you, plamaiziere.

Getting CARP set up is not the issue. Getting PF to redirect through a CARP interface is the issue.



> When I set the default gateway on server 1 to the carp0 interface, the PC cannot ssh to server 1. When I set the default gateway on server 1 to the lan0 interface, the PC can ssh to server 1.



Everything in this setup works as expected. We even have a CARP setup in a live environment in several locations for several months but no one seems to have an answer to the PF redirect issue.

We are going to roll back to our original setup using Nagios to monitor interfaces and Event Handlers to control Interface state.

Thanks for everyone's help on this.


----------

