# CARP setup Hetzner (vSwitch)



## nbari (Oct 27, 2021)

I am testing *CARP* in two dedicated server from Hetzner having both servers atached to a vSwitch with IPv4 addresses,  this is the what I have in /etc/rc.conf:


```
cloned_interfaces="vlan0"
create_args_vlan0="vlan 4000 vlandev em0 mtu 1400 fib 1"
ifconfig_vlan0="ether random"
```

I need a VLAN in order to configure the public IPv4 from the vSwitch, I can't set the VIP as an alias to the main NIC.

And in /etc/rc.local (the same on both servers only difference is that server 2 uses advskew 100)


```
ifconfig vlan0 inet 137.23.7.140/29 vhid 1 advskew 0 pass xxxyyyzzz 
route add default 137.23.7.137 -fib 1
sleep 3
if `ifconfig vlan0 | grep -q "carp: MASTER"`; then
    setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137
fi
```

I am using /etc.rc.local, because I need to change the mac address of the VLAN, and in case of rebooting the sever I just want to ping the gateway if CARP is "MASTER", but this is indeed my question/problem. (I had to change the mac address of the VLAN because otherwise the mac address of the main host will advertize into the vSwitch and this is not allowed by Hetzner)

The setup works fine, but after approximately 5 minutes the vSwithh stop routing packets to the server holding the VIP, what I notice is that periodically (cron every minute) I need to advertise the mac address of the server holding the VIP (`arping`) If I don't do this, the vSwitch stops routing packets.

Is there a way to periodically advertise the mac address? I tried without luck

```
sysctl net.link.ether.inet.max_age=60
```

In both servers I have also this in /etc/devd/carp.conf


```
notify 0 {
    match "system"      "CARP";
    match "subsystem"   "[0-9]+@[0-9a-z]+";
    match "type"        "(MASTER|BACKUP)";
    action "/root/carpcontrol.sh $subsystem $type | logger &";
};
```
 
This calls /root/carpcontrol.sh:


```
if [ $# -lt 2 ]; then
    echo 'missing arguments subsystem and (BACKUP|MASTER)'
    exit 1
fi

IP_GW="137.23.7.137"

# arping and ping to speed up the switch
if [ $2 = "MASTER" ]; then
    setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U ${IP_GW}
fi
```

It works fine, but the only If I keep using this cron:


```
*/1 * * * * ifconfig vlan0 | grep -q 'carp: MASTER' && setfib 1 /usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137 > /dev/null 2>&1
```

Wondering what could be wrong that I need to keep advertising/pinging and if there is a way to prevent this.

Any thoughts?


----------



## Jose (Oct 28, 2021)

nbari said:


> Is there a way to periodically advertise the mac address? I tried without luck...


This is exactly what CARP does. It periodically advertises a MAC address.



nbari said:


> The setup works fine, but after approximately 5 minutes the vSwithh stop routing packets to the server holding the VIP...


Try using tcpdump(1) to see what changes after 5 minutes.


----------



## covacat (Oct 28, 2021)

probably your vlan mac expires in gw's arp cache and it's request is not answered
do you have a route for 137.23.7.136/29 via vlan0 in fib0 ?
if not try to add one


----------



## nbari (Oct 28, 2021)

Jose said:


> This is exactly what CARP does. It periodically advertises a MAC address.
> 
> 
> Try using tcpdump(1) to see what changes after 5 minutes.


Hi, the provider told me that multicast and real multicast aren't possible inside the vSwitch for technical reasons and that all packets will be handled like broadcast inside the vSwitch.
What I notice via tcpdump after the vSwitch forgets the mac address are messages like:


```
f2:0b:a4:d1:20:01    Broadcast    ARP    56    Who has 137.23.7.140? Tell 137.23.7.137
```

I also notice that CARP is indeed announcing to address as expected to  224.0.0.18 (VRRP)  wondering if there could be a way to use like UNICAST like keepalive I think in where I could define the endpoint to send the announcements?


----------



## nbari (Oct 28, 2021)

covacat said:


> probably your vlan mac expires in gw's arp cache and it's request is not answered
> do you have a route for 137.23.7.136/29 via vlan0 in fib0 ?
> if not try to add one


Hi, I can't I get :


```
fib 0: Network is unreachable
```


----------



## covacat (Oct 28, 2021)

try route add -net  137.23.7.136/29 -interface vlan0 -fib 0


----------



## VladiBG (Oct 28, 2021)

CARP is based on multicast. If your provider doesn't support it then don't use CARP.


----------



## nbari (Oct 28, 2021)

VladiBG said:


> CARP is based on multicast. If your provider doesn't support it then don't use CARP.


Agree, but partially works if using arping periodically or using a private IP.

Any alternatives or suggestions? my goal is to configure an HAproxy in HA mode having a single VIP.


----------



## nbari (Oct 28, 2021)

covacat said:


> try route add -net  137.23.7.136/29 -interface vlan0 -fib 0


Thanks, that created the route but still not working, I need to call arping to make it work:


```
/usr/local/sbin/arping -i vlan0 -w 3 -U 137.23.7.137
```


----------



## covacat (Oct 28, 2021)

try without fibs first
just set a static route to some external box via 137.23.7.137 and see if that scenario works


----------



## nbari (Oct 28, 2021)

covacat said:


> try without fibs first
> just set a static route to some external box via 137.23.7.137 and see if that scenario works


I tried without luck:


```
route add 137.23.7.143 137.23.7.137 
and

route add 137.23.7.143 137.23.7.137 -fib 1
```

From my understanding having the routes should allow replying to the broadcast messages from the switch?


----------



## Jose (Oct 28, 2021)

VladiBG said:


> CARP is based on multicast. If your provider doesn't support it then don't use CARP.


Huh? CARP works by sending gratuitous ARP announcements at the link layer. These are hardware-level broadcasts.

There are some broken devices that don't handle these properly. I'm guessing that's the problem here:





						141023 – [carp] CARP arp replays with wrong src mac
					






					bugs.freebsd.org
				





			gratuitous ARP from CARP backup host


----------



## nbari (Oct 28, 2021)

Something that I notice, is that if I don't use *CARP* and just fix/configure the VIP, when running arp -an the mac is present as "vlan0 permanent" (I don't need to arping, it works fine without problems)


```
$ setfib 1 arp -na
? (137.23.7.137) at f2:0b:a4:d1:20:01 on vlan0 expires in 1199 seconds [vlan]
? (137.23.7.140) at e2:0c:ec:f6:f2:36 on vlan0 permanent [vlan]
```

But when using *CARP* this is the output, notice there is no entry for the VIP, only for the GW:


```
$ setfib 1 arp -na
? (137.23.7.137) at f2:0b:a4:d1:20:01 on vlan0 expires in 1199 seconds [vlan]
```

After running "arping -i vlan0 -w 3 -U 137.23.7.137", things start to work, this is  part of the tcpdump output:


```
e2:0c:ec:f6:f2:36     Broadcast    ARP    58    Who has 137.23.7.137? Tell 137.23.7.140 (duplicate use of 137.23.7.140 detected!)
`--- [Duplicate IP address detected for 137.23.7.140 (e2:0c:ec:f6:f2:36) - also in use by 00:00:5e:00:01:01 (frame 368)]
...

fe:6e:94:89:c7:c9    JuniperN_a5:f7:e4    ARP    42    137.23.7.140 is at 00:00:5e:00:01:01
```

Something interesting that I notice is that for some reason the MAC address that was advertised is 00:00:5e:00:01:01 but that is not the mac address of the vlan0 interface: e2:0c:ec:f6:f2:36 .  (this could explain why arping works since it is announcing the mac of the VLAN, but also because of that I get the duplicate use of <IP> detected!

Any idea about how to make CARP send the MAC address of the current interface used, in this case, a VLAN?


----------



## VladiBG (Oct 28, 2021)

Jose CARP announcements between the nodes are send using multicast. In pfsense you can overwrite it to use unicast by specifying the other node address. The announcement of the active (master) node toward the LAN segment is done using gratuitous arp. Other protocols like HSRP and VRRP works on the same principle.


----------



## nbari (Oct 28, 2021)

VladiBG said:


> Jose CARP announcements between the nodes are send using multicast. In pfsense you can overwrite it to use unicast by specifying the other node address. The announcement of the active (master) node toward the LAN segment is done using gratuitous arp. Other protocols like HSRP and VRRP works on the same principle.


How could this be done with pf ? or it is something unique to pfsense? 
Currently, I am using arping to do the gratuitous ARP, and pf* route-to*  because I have the VIP in another route (fib 1) something like this:


```
pass in on vlan0 reply-to (vlan0 137.23.7.137) inet proto tcp from any to any port = https flags S/SA keep state
```

So far is working,  but I would like to have a cleaner setup to get rid mainly of the "duplicated mac addresses message" and the arping that I have in a cron


----------



## Jose (Oct 29, 2021)

VladiBG said:


> Jose CARP announcements between the nodes are send using multicast.


You're absolutely right. I was being ignorant. The CARP advertisements are multicast on the local subnet. Details on this are hard to find. This is the best explanation I found, and it's for Openbsd CARP:








						OpenBSD CARP Notes
					

CARP is the Common Address Redundancy Protocol. It's a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol. CARP was created and is maintained by the OpenBSD project. The notes here apply to OpenBSD 5.0 and higher. Protocol Information Virtual...




					www.packetmischief.ca
				




CARP absolutely will not work without multicast. In its absence, both nodes will think they're the master because they can't "hear" the backup, and will send gratuitous ARP to grab the VIP. This explains the duplicate IP address messages Nbari is seeing. Both nodes are trying to grab the VIP at the same time. I'm not sure how ARP handles conflicts like this. It might be undefined behavior.


----------



## VladiBG (Oct 29, 2021)

nbari said:


> How could this be done with pf ?


It's not about the firewall (pf) it's how the CARP is modified in pfsense. There's the same discussion from 2020 on netgate forum regarding the vSwitch and CARP.








						CARP issues due to Layer 2 switching
					

Hi all, I am having an issue with CARP using Hetzner's vSwitch to connect servers across different DC's. As CARP wasn't working out of the box, talked to them and they replied this:  Real Multicast isn't possible inside the vSwitch. All packets will be ha...




					forum.netgate.com
				




If the vSwitch handles gARP properly but not allows multicast then you need another way to connect the nodes together so they can talk to each other. Maybe with gif or gre tunnels between them.

Here's another example from OpenBSD and _syncpeer _:


			OpenBSD PF: Firewall Redundancy (CARP and pfsync)


----------



## nbari (Oct 29, 2021)

Jose said:


> You're absolutely right. I was being ignorant. The CARP advertisements are multicast on the local subnet. Details on this are hard to find. This is the best explanation I found, and it's for Openbsd CARP:
> 
> 
> 
> ...


Interesting that in my case they do work, indeed one server is in BACKUP and the other as MASTER, also when doing:


```
ifconfig vlan0 vhid 1 state master
or
ifconfig vlan0 vhid 1 state backup
```

They work as expected so I don't know if this is something within the vSwitch that partially supports it or the CARP states are only broadcasted.


----------



## nbari (Oct 29, 2021)

multicast is blocked in many environments, therefore wondering if there is another alternative or if implementing something on top of broadcast could work? just wondering if when broadcasting one can send the vhid, password, etc, some data that could give room to create/start drafting something, ideally this should be at layer 2 but maybe something interesting for starting a project.


----------



## covacat (Oct 29, 2021)

from what i gathered incoming packets should come to the virtual carp mac 00:00:5e:00:01:VHID and  outgoing should have the real nic mac addr and this may confuse/or be denied in some configuration
i don't have a carp install to test


----------



## nbari (Oct 29, 2021)

I only see the VLAN mac address (when using ifconfig) but when using tcpdump I notice 00:00:5e:00:01:VHID


----------



## nbari (Oct 29, 2021)

VladiBG said:


> It's not about the firewall (pf) it's how the CARP is modified in pfsense. There's the same discussion from 2020 on netgate forum regarding the vSwitch and CARP.
> 
> 
> 
> ...


I created a gif/gre tunnel, which works but I can't find a way to add an alias or use a tap interface so that I could use CARP on it, any ideas?

In both cases I get:

```
ifconfig: SIOCGVH: Operation not supported
```


----------



## VladiBG (Oct 29, 2021)

Getting CARP to broadcast on a different interface


----------



## nbari (Oct 29, 2021)

For some reason, the TAP interface seems to be always down, and CARP remains in INIT


----------



## rim (Mar 26, 2022)

Patch to fix src mac for arp traffic: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262828


----------

