# Redirection of connections to local CARP interface



## feffe (Jun 27, 2013)

Hi,

The short version:

Is it possible to create NAT and redirect rules for connections initialized on the local machine to an IP address on a local CARP interface?

Longer version and background:

For various reasons I need to set[]up a fairly complicated clustered SMTP service. One layer of this cake is a combined load balancer and spamfilter. Once the mail goes through the spam filter they should be relayed to a single CARP:ed IP address (let's call it $carp_ip). This $carp_ip is configured so that any of the machines in this layer can "own" the address.

I set up the following rules in pf.conf to NAT and redirect connections to $carp_ip to the SMTP nodes in the next layer:

```
rdr on $int_if inet proto tcp from any to $carp_ip port 25 -> <smtp_nodes>
nat on $int_if inet proto tcp from $local_net to <smtp_nodes> port 25 -> $carp_ip
...
pass in quick on $int_if inet proto tcp from any to $carp_ip port 25
```

I tested by using telnet from one of the server that were not CARP master, which connects just fine. However, when I try to connect with telnet from the CARP master I simply got a "Connection refused".

Figuring that was kind of odd, I fired up my trusted tcpdump and found that telnet did not send the request on $int_if, or even $carp_if, but on the loopback device:

```
# tcpdump -ni lo0 tcp port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes
16:43:00.581612 IP 10.0.0.2.62291 > 10.0.0.5.25: Flags [S], seq 2093852831, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 25231064 ecr 0], length 0
16:43:00.581649 IP 10.0.0.5.25 > 10.0.0.2.62291: Flags [R.], seq 0, ack 2093852832, win 0, length 0
```

Well, that's odd, so figuring there was just one interface I didn't want this _NAT_/redirect rules to apply for I simply changed the rules to this:

```
rdr on !$ext_if inet proto tcp from any to $carp_ip port 25 -> <smtp_nodes>
nat on !$ext_if inet proto tcp from $local_net to <smtp_nodes> port 25 -> $carp_ip
...
pass in quick on !$ext_if inet proto tcp from any to $carp_ip port 25
```

Now telnet does not respond with "Connection refused", it just times out after a while. tcpdump on lo0 tells me that it sends a request, but receives no reply. But now interesting things happen on $int_if:


```
# tcpdump -ni $int_if tcp port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on $int_if, link-type EN10MB (Ethernet), capture size 65535 bytes
16:50:20.503866 IP 10.0.0.5.56150 > 10.0.0.73.25: Flags [S], seq 3129724327, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 25670986 ecr 0], length 0
16:50:20.503966 IP 10.0.0.73.25 > 10.0.0.5.56150: Flags [S.], seq 940845167, ack 3129724328, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2075520144 ecr 25670986], length 0
16:50:20.503979 IP 10.0.0.5.56150 > 10.0.0.73.25: Flags [R], seq 3129724328, win 0, length 0
```

So the redirect and NAT rule works as expected, but the reply is never returned to telnet which, for some reason, sends the request on lo0 (even when a specific source address is specified)... Or something?

So could anyone explain what's going on here? And it is even possible to do what I want?


----------

