# PF + gif + ipsec + racoon + routing problems



## ouido (May 11, 2013)

Hello,

I am experiencing some difficulty with a site-to-site VPN tunnel between two private networks, each one behind a FreeBSD 8.3 + PF router/firewall.  The layout is like most:

(networkA) <-> (gatewayA) <- gif+ipsec tunnel -> (gatewayB) <-> (networkB)

Both gateways are mature, tuned systems that are working flawlessly for routing and NAT on behalf of their networks.

I've done the preliminary setup of the tunnel following the guidelines in the Handbook (http://www.freebsd.org/doc/en/books/handbook/ipsec.html).  Once it was working, I studied a lot more, tested to make sure encryption was working, and hardened the rules.

For discussion purposes, let's assume that networkA is 1.1.1.0/24 and networkB is 2.2.2.0/24, with each gateway sitting at the .254 address (1.1.1.254, 2.2.2.254).  The public addresses are A.A.A.A and B.B.B.B.  As such, the gif tunnels are configured as follows:

Setting up the tunnels, routing, etc.

gatewayA:

```
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
	tunnel inet A.A.A.A --> B.B.B.B
	inet 1.1.1.254 --> 2.2.2.254 netmask 0xffffff00 
	options=1<ACCEPT_REV_ETHIP_VER>
```

gatewayB:

```
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
	tunnel inet B.B.B.B --> A.A.A.A
	inet 2.2.2.254 --> 1.1.1.254 netmask 0xffffff00 
	options=1<ACCEPT_REV_ETHIP_VER>
```

Static routes have been created on each machine: `route add -net 1.1.1.0/24 1.1.1.254` and `route add -net 2.2.2.0/24 2.2.2.254` respectively.  Appropriate pf rules have been created on each end to appoint the gif0 interface as a trusted one (i.e. `pass quick on gif0 label "pass vpn"`).  At this point, with no ipsec encryption enabled, all traffic flows perfectly between the two gateways and their respective networks (i.e. the gateways can ping each other, each other's hosts, and hosts can ping each other across the networks as well).  I emphasize this point just to ensure everyone that the fundamental networking is sound and that I've checked all of those aspects thoroughly.

Introducing encryption

For the sake of brevity, I will not repeat the Handbook article with respect to this step.  Let's just say that I've followed it closely and performed the advised testing.  Since the /usr/local/etc/ipsec.conf file is the subject of my post today, I will cite it for you here:

gatewayA:

```
flush;
spdflush;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/use;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/use;
```

gatewayB:

```
flush;
spdflush;
spdadd 2.2.2.0/24 1.1.1.0/24 any -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/use;
spdadd 1.1.1.0/24 2.2.2.0/24 any -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/use;
```

First test case: After starting up the racoon daemon, setting the policies, etc. I am able to ping between the two gateways.  The ESP session is established and a `tcpdump -i em0 host A.A.A.A and host B.B.B.B` on each machine verifies that the inbound and outbound packets are encrypted.  A `tcpdump -i gif0` shows the decrypted ping packets as hoped.  So far so good.

Next, we test a ping from gatewayB to a machine behind gatewayA.  The pings appear to work, however the running tcpdump output on gatewayB's em0 interface clearly shows that the entire connection is not encrypted, like this:

```
19:17:37.871106 IP B.B.B.B > A.A.A.A: ESP(spi=0x0e639454,seq=0x84), length 116
19:17:37.871460 IP A.A.A.A > B.B.B.B: IP 1.1.1.1 > 2.2.2.254: ICMP echo reply, id 52418, seq 49, length 64 (ipip-proto-4)
```

Workaround

If we change our /usr/local/etc/ipsec.conf files to match packets on the public IPs + ipencap, we get full encryption as desired, like this:

gatewayA:

```
flush;
spdflush;
spdadd A.A.A.A/32 B.B.B.B/32 ipencap -P out ipsec esp/tunnel/A.A.A.A-B.B.B.B/use;
spdadd B.B.B.B/32 A.A.A.A/32 ipencap P in ipsec esp/tunnel/B.B.B.B-A.A.A.A/use;
```

gatewayB:

```
flush;
spdflush;
spdadd B.B.B.B/32 A.A.A.A/32 ipencap -P out ipsec esp/tunnel/B.B.B.B-A.A.A.A/use;
spdadd A.A.A.A/32 B.B.B.B/32 ipencap -P in ipsec esp/tunnel/A.A.A.A-B.B.B.B/use;
```

Summary

It seems that when the packets are routed in to the gateway from their local network hosts, the src and dst addresses are changed to the public IPs of the tunnel -- at least from the perspective of the ipsec stack.  While the workaround is fine for scenarios where I control both endpoints, I am trying to integrate a remote endpoint (i.e. Cisco ASA) where I cannot get them to implement an equivalent workaround on their end.  Does anyone have any ideas that might help me get ipsec to properly match off of the private src and dst addresses?


----------



## ouido (May 21, 2013)

On a stroke of good luck, I found the culprit in /etc/sysctl.conf:

```
net.inet.ip.fastforwarding=1
```
When enabled, this breaks IPsec per my post above.  When disabled (default), IPsec works as it should as described in the FreeBSD handbook.


----------



## megapearl (Feb 6, 2015)

Yes! you did the trick! I've searched for ages and never found the solution until I found your post! Thanks!!


----------



## junovitch@ (Feb 8, 2015)

It does seem like it's kind of buried but it is mentioned in inet(4)

```
IPCTL_FASTFORWARDING  (ip.fastforwarding) Boolean: enable/disable the use
                           of fast IP forwarding code.  Defaults to off.  When
                           fast IP forwarding is enabled, IP packets are for-
                           warded directly to the appropriate network inter-
                           face with direct processing to completion, which
                           greatly improves the throughput.  All packets for
                           local IP addresses, non-unicast, or with IP options
                           are handled by the normal IP input processing path.
                           All features of the normal (slow) IP forwarding
                           path are supported including firewall (through
                           [MAN=9]pfil[/MAN] hooks) checking, except [MAN=4]ipsec[/MAN] tunnel
                           brokering.  The IP fastforwarding path does not
                           generate ICMP redirect or source quench messages.
```


----------

