# IPv6 connectivity lost after some time



## basbebe (Jul 19, 2017)

Since a few weeks I'm having a problem with the IPv6 connectivity of my FreeBSD 11.0 server. There had been no software updates or configuration changes that I know of that could cause the problem.

The server only has IPv6 connectivity (incoming and outgoing) for a short time (approx. 40-80 minutes) after a reboot. Then it is gone and I don't know why.

It worked like a charm for months before.

This is my config:

`/etc/rc.conf`


```
#IPv4

ifconfig_vtnet0_name="em0"
ifconfig_em0="inet YY.YY.YY.YY netmask 255.255.252.0"
defaultrouter="188.TT.TT.TT"

# IPv6

ifconfig_em0_ipv6="inet6 accept_rtadv"
rtsold_enable="YES"
ifconfig_em0_alias0="inet6 2aXX:XXXX:XXXX:XXXX::1 prefixlen 64"
ipv6_defaultrouter="fe80::1%em0"

gateway_enable="YES"
ipv6_gateway_enable="YES"
```

`ifconfig`


```
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ZZ:ZZ:ZZ:ZZ:ZZ:ZZ
    inet YY.YY.YY.YY netmask 0xfffffc00 broadcast 188.DD.DD.DD
    inet6 fe80::KKKK:KKKK:KKKK:1822%em0 prefixlen 64 scopeid 0x1
    inet6 2aXX:XXXX:XXXX:XXXX::1 prefixlen 64
    inet6 2aXX:XXXX:XXXX:XXXX::103 prefixlen 128
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T <full-duplex>
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    inet 10.0.0.1 netmask 0xffffffff
    inet 10.0.0.4 netmask 0xffffffff
    inet 10.0.0.3 netmask 0xffffffff
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
```

The routing tables stay the same after the fail:

`netstat -rn`


```
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            188.TT.TT.TT        UGS         em0
10.0.0.1           link#2             UH          lo0
10.0.0.2           link#2             UH          lo0
10.0.0.3           link#2             UH          lo0
10.0.0.4           link#2             UH          lo0
10.0.0.10          link#2             UH          lo0
10.0.0.11          link#2             UH          lo0
10.0.0.12          link#2             UH          lo0
10.0.0.13          link#2             UH          lo0
127.0.0.1          link#2             UH          lo0
188.TT.TT.0/22     link#1             U           em0
YY.YY.YY.YY       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           fe80::1%em0                   UGS         em0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2aXX:XXXX:XXXX:XXXX::/64             link#1                        U           em0
2aXX:XXXX:XXXX:XXXX::1               link#1                        UHS         lo0
2aXX:XXXX:XXXX:XXXX::102             link#1                        UHS         lo0
2aXX:XXXX:XXXX:XXXX::103             link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::KKKK:KKKK:KKKK:1822%em0     link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
```


What could be causing this?
Any logs I should check? Anything I might be missing?
Thanks!


----------



## SirDice (Jul 19, 2017)

basbebe said:


> ```
> ifconfig_em0_ipv6="inet6 accept_rtadv"
> rtsold_enable="YES"
> ifconfig_em0_alias0="inet6 2aXX:XXXX:XXXX:XXXX::1 prefixlen 64"
> ...


You typically use SLAAC or static addresses, not both.

Also note the discrepancy:

```
inet6 2aXX:XXXX:XXXX:XXXX::1 prefixlen 64
    inet6 2aXX:XXXX:XXXX:XXXX::103 prefixlen 128
```
The SLAAC received address has a prefixlen of 128 and your static address has 64.


----------



## basbebe (Jul 19, 2017)

SirDice said:


> You typically use SLAAC or static addresses, not both.
> 
> Also note the discrepancy:
> 
> ...



Thanks for pointing this out.

This does not work, however:

`/etc/rc.conf`

```
#IPv4

ifconfig_vtnet0_name="em0"
ifconfig_em0="inet YY.YY.YY.YY netmask 255.255.252.0"
defaultrouter="188.TT.TT.TT"

# IPv6


rtsold_enable="YES"
ifconfig_em0_ipv6="inet6 2aXX:XXXX:XXXX:XXXX::1 prefixlen 64"
ipv6_defaultrouter="fe80::1%em0"

gateway_enable="YES"
ipv6_gateway_enable="YES"
```

After a restart there is no IPv6 connectivity. Ifconfig and netstat seem to show the same results like before (except the missing `ACCEPT_RTADV`) but the server is unreachable from the outside and I can neither `ping6` nor `traceroute6` from the inside.

The ::102 and ::103 addresses are addresses of jails that got added by ezjail's config.


----------



## SirDice (Jul 19, 2017)

Settings look fine but I have no way of knowing what your provider uses. So I would suggest asking them for the correct settings. At the very least you should be able to ping the default gateway, if that doesn't work there's something wrong with addresses you're using.


----------



## basbebe (Jul 19, 2017)

SirDice said:


> Settings look fine but I have no way of knowing what your provider uses. So I would suggest asking them for the correct settings. At the very least you should be able to ping the default gateway, if that doesn't work there's something wrong with addresses you're using.


Pinging the local Gateway (fe80::1) always works, also`traceroute6 fe80::1%em0`.
But anything else doesn't work.
As I said, this configuration worked fine for over a year (FreeBSD 10.x and 11.0) and now I keep having this problems.
The Provider says he doesn't know of any problems and that it must be an error in my server's config…

Disabling pf doesn't change anything as well.

Could adding a static route change anything?
Since it worked the way it did I don't know where to start.

And I have no cron jobs on the host system besides the default ones (newsy slog, atrium, etc) and a weekly check for backups…

Pinging fe80::KKKK:KKKK:KKKK:1822 doesn't work as well by the way (after IPv6 stops working)


----------



## SirDice (Jul 19, 2017)

basbebe said:


> The Provider says he doesn't know of any problems and that it must be an error in my server's config…


Don't ask them if there are problems, ask them what the correct settings should be. Then compare that info with the information you've been using.


----------



## basbebe (Jul 19, 2017)

SirDice said:


> Don't ask them if there are problems, ask them what the correct settings should be. Then compare that info with the information you've been using.


The information they provide (my IPs, my Gateway) haven't changed and are correct (fe80::1)


----------



## SirDice (Jul 19, 2017)

basbebe said:


> Pinging the local Gateway (fe80::1) always works, also `traceroute6 fe80::1%em0`.


Does a `traceroute6 -n 2001:4860:4860::8888` not go any further than fe80::1? If that's the case the issue is probably with the provider. If you can ping/traceroute the default gateway your setup is correct and issue is somewhere beyond the gateway (and on the provider's network).


----------



## basbebe (Jul 19, 2017)

SirDice said:


> Does a `traceroute6 -n 2001:4860:4860::8888` not go any further than fe80::1? If that's the case the issue is probably with the provider. If you can ping/traceroute the default gateway your setup is correct and issue is somewhere beyond the gateway (and on the provider's network).



trace routes after IPv6 stopped working don't give any answer at all:


```
❯ traceroute6 -n 2001:4860:4860::8888                                                                                                                                                    ⏎
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2aXX:XXXX:XXXX:XXXX::1, 64 hops max, 12 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
```


----------



## SirDice (Jul 19, 2017)

In hindsight that might not be a good test address, I'm not getting anything on 2001:4860:4860::8888 either, I'm not even getting a response from anything in between:

```
root@armitage:~ # traceroute6 -n 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:7c8:aaaf:2dd::1, 64 hops max, 12 byte packets
(nothing happening)
```
Ping works though:

```
root@armitage:~ # ping6 -c4 2001:4860:4860::8888
PING6(56=40+8+8 bytes) 2a01:7c8:aaaf:2dd::1 --> 2001:4860:4860::8888
16 bytes from 2001:4860:4860::8888, icmp_seq=0 hlim=55 time=4.852 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=1 hlim=55 time=5.128 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=2 hlim=55 time=4.901 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=3 hlim=55 time=5.308 ms

--- 2001:4860:4860::8888 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 4.852/5.047/5.308/0.183 ms
```
And I don't have any IPv6 related issues.


----------



## basbebe (Jul 19, 2017)

Just after a restart it works (as well as ipv6.google.com):


```
~ # ❯❯❯ traceroute6 -n 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2aXX:XXXX:XXXX:XXXX::1 , 64 hops max, 12 byte packets
 1  2a03:4000:6:f000::2  0.530 ms  0.174 ms  0.629 ms
 2  2a03:4000:0:10::3  3.980 ms  18.043 ms  4.051 ms
 3  2a00:11c0::1:0:0:371  4.215 ms  4.222 ms  4.161 ms
 4  2001:7f8::3b41:0:2  4.548 ms  4.638 ms  4.560 ms
 5  2001:4860::1:0:d0c6  4.889 ms
    2001:4860::1:0:d0d9  4.703 ms
    2001:4860::1:0:d0c5  4.620 ms
 6  2001:4860::8:0:cb95  11.580 ms
    2001:4860::c:4000:f874  5.021 ms
    2001:4860::8:0:cb95  11.342 ms
 7  2001:4860::8:4000:d325  11.570 ms
    2001:4860::8:0:cc40  11.383 ms
    2001:4860::8:4000:d324  30.013 ms
 8  2001:4860::2:0:a94a  12.771 ms
    2001:4860::2:0:76e8  11.404 ms
    2001:4860::2:0:76e7  11.364 ms
 9  * * *
10  *

~ # ❯❯❯ ping6 -c4 2001:4860:4860::8888
PING6(56=40+8+8 bytes) 2aXX:XXXX:XXXX:XXXX::1 --> 2001:4860:4860::8888
16 bytes from 2001:4860:4860::8888, icmp_seq=0 hlim=50 time=11.315 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=1 hlim=50 time=11.456 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=2 hlim=50 time=11.404 ms
16 bytes from 2001:4860:4860::8888, icmp_seq=3 hlim=50 time=12.084 ms

--- 2001:4860:4860::8888 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 11.315/11.565/12.084/0.304 ms

~ # ❯❯❯ ping6 -c 5 ipv6.google.com
PING6(56=40+8+8 bytes) 2aXX:XXXX:XXXX:XXXX::1 --> 2a00:1450:4001:81e::200e
16 bytes from 2a00:1450:4001:81e::200e, icmp_seq=0 hlim=58 time=4.576 ms
16 bytes from 2a00:1450:4001:81e::200e, icmp_seq=1 hlim=58 time=4.562 ms
16 bytes from 2a00:1450:4001:81e::200e, icmp_seq=2 hlim=58 time=4.479 ms
16 bytes from 2a00:1450:4001:81e::200e, icmp_seq=3 hlim=58 time=4.572 ms
16 bytes from 2a00:1450:4001:81e::200e, icmp_seq=4 hlim=58 time=4.796 ms

--- ipv6.l.google.com ping6 statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 4.479/4.597/4.796/0.106 ms

~ # ❯❯❯ traceroute6 -n ipv6.google.com
traceroute6 to ipv6.l.google.com (2a00:1450:4001:81e::200e) from 2aXX:XXXX:XXXX:XXXX::1 , 64 hops max, 12 byte packets
 1  2a03:4000:6:f000::2  0.245 ms  0.215 ms  0.244 ms
 2  2a03:4000:0:10::3  19.065 ms  4.109 ms  29.701 ms
 3  2a00:11c0::1:0:0:371  4.214 ms  4.126 ms  4.166 ms
 4  2001:7f8::3b41:0:2  4.626 ms  4.465 ms  4.432 ms
 5  2001:4860:0:11e0::1  4.796 ms  4.694 ms
    2001:4860:0:11e2::1  4.785 ms
 6  2001:4860:0:1::2043  4.603 ms
    2001:4860:0:1::2041  4.566 ms
    2001:4860:0:1::2043  4.480 ms
 7  2a00:1450:4001:81e::200e  4.428 ms  4.476 ms  4.475 ms
```


----------



## basbebe (Jul 19, 2017)

and after it doesn't:


```
~ # ❯❯❯ traceroute6 -n 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2aXX:XXXX:XXXX:XXXX::1 , 64 hops max, 12 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *


~ # ❯❯❯ ping6 -c4 2001:4860:4860::8888                                                                                                                                               ⏎
PING6(56=40+8+8 bytes) 2aXX:XXXX:XXXX:XXXX::1 --> 2001:4860:4860::8888

--- 2001:4860:4860::8888 ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
```


----------



## basbebe (Jul 19, 2017)

Provider says that if I still can access the gateway, my system seems to loose the route and if it was a problem in their infrastructure then I couldn't access the gateway at all.


----------



## SirDice (Jul 19, 2017)

That's a really bad analysis. If you can access the gateway routing is obviously not a problem on your end. The only way to lose a default route is when the interface goes down (which I presume doesn't happen). And just because you can access the gateway it doesn't mean you can access anything beyond that gateway. So their infrastucture could still be the problem (anything beyond the gateway is part of their infrastructure).


----------



## cwf-ml (Jul 28, 2017)

basbebe said:


> Provider says that if I still can access the gateway, my system seems to loose the route and if it was a problem in their infrastructure then I couldn't access the gateway at all.


Not from their POV. Presumable basbebe isn't the only one using the gateway, and the provider is monitoring both the gateway and its internet link. So from their PoV, the gateway is green and once basbebe gets there, he or she should be green as well. 

Issues to look at
- are you really loosing that route?
- what about neigbour discovery? 
- why run a router solicitation demon if you don't need it. What does it do?
- what default route did the provider give you? On default routes there is the fe80 school of thought, and the use the real address school. I admit preferring the latter, did you try that?

We do run a few older BSD servers with route solicitation, but we needed major testing to find a reliable config for all components involved. These days people use hard configured addresses and hsrp/vrrp. Or use real routing protocols fore more complex situations (quagga and ospfv3/ospf6 work just fine on FreeBSD) -  but that would be overkill for a "just find the uplink" situation.


----------

