# Intermittent ICMP6 destination unreachable



## FUNTOWNE (Sep 7, 2018)

Hi All -

I am a bit of an IPv6 novice, so if anything below is off, please do not hesitate to show where I could improve how I describe something and/or fix a dumb error.

*The problem and current status*:

I have been working on and off to configure my OpenVPN server to handle both IPv6 and IPv4 on my VPS, which has one virtual internet interface, a static IPv4 IP and a static IPv6 prefix.  The current status is that it more or less works, but has the following wonky behaviors:

- When monitoring icmp6 traffic via tcpdump (tcpdump -n -i vtnet0 icmp6), rougly every other attempt to `ping6 google.com` returns the following:
`fe80::fc54:ff:fe34:aaaa > fe80::5054:ff:fe34:aaaa: ICMP6, destination unreachable, beyond scope 2607:f8b0:4009:80b::200e, source address fe80::5054:ff:fe34:aaaa, length 64` -- fe80::5054:ff:fe34:aaaa is my gateway (I think? or is it a link-local?), more on that below.  I have no idea why my system thinks that its IP address is the gateway's address..?  This is the one that really stumps me.

- When the attempt to ping6 google.com succeeds (more often than not), it looks like normal ping6 traffic:
`IP6 2001:1111:2222:3333:4444:aaaa:bbbb:cccc > 2607:f8b0:4009:80b::200e: ICMP6, echo request, seq 9, length 16`

- Clients utilizing IPv6 over my VPN see the VPN apparently round robin outbound traffic through my IPv6 addresses (testing via sites like ipv6-test.com test-ipv6.com and internet.nl).  I also see intermittent failures which I believe are related to the behavior of the first point above.  Otherwise, IPv6 browsing seems more or less reliable for VPN clients, less so for the host machine.  Maybe I am misinterpreting the 'prefer_source' and 'deprecated' flags' behavior, or am missing some sysctl values to alter this particular behavior?


*How I got here*:

To save a bunch of narrative, please find my relevant config files below that are creating this behavior:


rc.conf IPv6 bits

```
ipv6_ifconfig_vtnet0="2001:1111:2222:3333:4444:aaaa:bbbb:cccc prefixlen 64 prefer_source"
ipv6_ifconfig_vtnet0_alias0="2001:1111:2222:3333:4444:dddd:eeee:ffff prefixlen 64 deprecated"
ipv6_cpe_wanif="vtnet0"
ipv6_defaultrouter="fe80::fc54:ff:fe34:aaaa%vtnet0"
ipv6_activate_all_interfaces="YES"
ipv6_gateway_enable="YES"
```

2001:1111:2222:3333:4444:aaaa:bbbb:cccc  is what I use for all outbound IPv6 traffic from the host.
2001:1111:2222:3333:4444:dddd:eeee:ffff is the reverse DNS entry for my domains + where my webserver lives; I don't want to have outbound non-web-server IPv6 traffic from this IP for privacy reasons.
2001:1111:2222:3333:4444:ffff:ffff:ffff (below) is assigned by my VPS provider

ifconfig output

```
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether aa:ff:ff:34:aa:aa
    hwaddr aa:ff:ff:34:aa:aa
    inet 88.88.88.88 netmask 0xffffff00 broadcast 88.88.88.255 [not real of course!]
    inet6 fe80::5054:ff:fe34:aaaa%vtnet0 prefixlen 64 scopeid 0x1
    inet6 2001:1111:2222:3333:4444:aaaa:bbbb:cccc  prefixlen 64 prefer_source
    inet6 2001:1111:2222:3333:4444:dddd:eeee:ffff prefixlen 64 deprecated
    inet6 2001:1111:2222:3333:4444:ffff:ffff:ffff prefixlen 64 autoconf
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
```

Yes, I know my pf.conf is a bit sloppy and some rules may be redundant... such is learning. I want to focus on the problems above first vs making the file tidier 

pf.conf

```
ext_if=vtnet0
tcp_pass= "{ 22225 80 443 27015 10001 4004 3128 }"
udp_pass= "{ 22225 80 443 27015 10001 4004 3128 }"
vpnclients = "192.168.88.0/24"
vpnclients6 = " 2001:1111:2222:3333:5555::/65"
vpnint = "tun0"
set optimization aggressive
set block-policy return
set skip on lo0
set skip on tun0
scrub all no-df random-id
scrub out all random-id no-df
nat on vtnet0 inet from $vpnclients to any -> vtnet0
nat on vtnet0 inet6 from $vpnclients6 to any -> vtnet0
block in all
pass out quick all keep state
table <sshguard> persist
block in log quick on $ext_if proto tcp from <sshguard> to any port 22225 label "ssh bruteforce 22225"
block in log quick on $ext_if proto tcp from <sshguard> to any port 3128 label "ssh bruteforce 3128"
block in log quick on $ext_if proto tcp from <sshguard> to any port 22 label "ssh bruteforce 22"
block in log quick on $ext_if proto tcp from <sshguard> to any port 2222 label "ssh bruteforce 2222"
block quick proto tcp from $ext_if to any port 25
pass in on vtnet0 proto tcp to port $tcp_pass keep state
pass in on vtnet0 proto udp to port $udp_pass keep state
pass in on vtnet0 inet6 proto tcp to port $tcp_pass keep state
pass in on vtnet0 inet6  proto udp to port $udp_pass keep state
pass in proto ipv6-icmp from any to any
pass in on $vpnint inet from any to any
pass in on $vpnint inet6 from any to any
pass in on vpnclients6 from any to any
pass in on vpnclients from any to any
```

openvpn.conf

```
port 10001
proto udp
proto udp6 #also likely redundant
dev tun
tun-ipv6 #I can probably remove this
ca   /usr/local/etc/openvpn/ca.crt
cert /usr/local/etc/openvpn/server.crt
key  /usr/local/etc/openvpn/server.key
dh   /usr/local/etc/openvpn/dh4096.pem
topology subnet
server 192.168.88.0 255.255.255.0
server-ipv6 2001:1111:2222:3333:5555::/65
ifconfig-ipv6 2001:1111:2222:3333:5555::1000 2001:1111:2222:3333:5555::1999
comp-lzo no
ecdh-curve secp384r1
push "redirect-gateway def1 bypass-dhcp"
push "redirect-gateway ipv6"
push "dhcp-option DNS 192.168.88.1"
push "comp-lzo no"
push "route-ipv6 2001:1111:2222:3333::/64"
push "route-ipv6 2000::/3"
duplicate-cn
keepalive 10 120
tls-crypt ta.key
persist-key
persist-tun
reneg-sec 0
cipher AES-256-CBC
auth sha512
tls-server
tls-version-min 1.2
user nobody
group nobody
persist-key
persist-tun
sndbuf 0
rcvbuf 0
verb 0
```

sysctl.conf IPv6-relevant bits

```
All defaults
```

Any help would be appreciated.  It took me a fair while to get to this point; I'm stumped at how to resolve the problems I have highlighted above.  Thanks in advance!

After a bit of googling, it looks like my NAT rule is (round robin) taking the link-local IP into consideration for routing.  That's what I need to avoid and can't figure out how.  Example: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199489


----------



## FUNTOWNE (Sep 7, 2018)

Typing this out and the bugzilla link netted a fix 

The offending NAT rule: `nat on vtnet0 inet6 from $vpnclients6 to any -> vtnet0`

Changed to `nat on vtnet0 inet6 from $vpnclients6 to any -> 2001:1111:2222:3333:4444:aaaa:bbbb:cccc`

Has fixed it.  I am sure there's still something not perfect with the above, but performance is now as expected.  pf appeared to be using the link-local when considering the nat rule, hard coding the rule to go out via my preferred address resolved the local ping / IPv6 Internet issue and intermittent VPN wierdness in one go.


----------

