# jail cause mac address change (flip-floping)



## crypt47 (Jan 17, 2019)

Hello, I have made two jailed vms (jid 1235 and 1236) using epair interface and bridged that to physical. I have very limited FreeBSD experience (but growing interest) and extended Linux experience. Hence did it in the same manner I would do it on Linux. Please look at some outputs of the testing stand (bge1 unused):


```
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                           
        options=80099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>                                                                                                   
        ether 00:1c:c4:81:bf:c2                                                                                                                                               
        inet 192.168.0.160 netmask 0xffffff00 broadcast 192.168.0.255                                                                                                         
        media: Ethernet autoselect (1000baseT <full-duplex,master>)                                                                                                           
        status: active                                                                                                                                                       
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                 
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                                               
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>                                                                                           
        ether 00:1c:c4:81:bf:c3                                                                                                                                               
        media: Ethernet autoselect                                                                                                                                           
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384                                                                                                             
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>                                                                                                       
        inet6 ::1 prefixlen 128                                                                                                                                               
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3                                                                                                                           
        inet 127.0.0.1 netmask 0xff000000                                                                                                                                     
        groups: lo                                                                                                                                                           
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                                                                                                             
ab0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                                     
        ether 02:d1:88:e6:66:00                                                                                                                                               
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15                                                                                                           
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                                                                                                             
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                                                                                                             
        member: epair1236a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                     
                ifmaxaddr 0 port 6 priority 128 path cost 2000                                                                                                               
        member: epair1235a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                     
                ifmaxaddr 0 port 5 priority 128 path cost 2000                                                                                                               
        member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                                                                                                           
                ifmaxaddr 0 port 1 priority 128 path cost 55                                                                                                                 
        groups: bridge                                                                                                                                                       
        nd6 options=9<PERFORMNUD,IFDISABLED>                                                                                                                                 
epair1235a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                     
        options=8<VLAN_MTU>                                                                                                                                                   
        ether 02:6f:e4:30:b7:0a                                                                                                                                               
        inet6 fe80::6f:e4ff:fe30:b70a%epair1235a prefixlen 64 scopeid 0x5                                                                                                     
        groups: epair                                                                                                                                                         
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                                                                                                   
        status: active                                                                                                                                                       
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                                                                                                             
epair1236a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                     
        options=8<VLAN_MTU>                                                                                                                                                   
        ether 02:9d:d1:69:c1:0a                                                                                                                                               
        inet6 fe80::9d:d1ff:fe69:c10a%epair1236a prefixlen 64 scopeid 0x6                                                                                                     
        groups: epair                                                                                                                                                         
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                                                                                                                   
        status: active                                                                                                                                                       
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

The problem is I get constant messages about mac change between epair and bridge interface:


```
arp: 192.168.0.160 moved from 00:1c:c4:81:bf:c2 to 02:6f:e4:30:b7:0a on epair1235b                                                                                            
arp: 192.168.0.160 moved from 02:6f:e4:30:b7:0a to 00:1c:c4:81:bf:c2 on epair1235b
```

Mainly this happens for one of the vms which has more network activity.
I really need some way to stop this behaviour (not to hide the log message!).
Any explanation of why FreeBSD is doing this is appreciated!


----------



## crypt47 (Jan 17, 2019)

Well, the solution seems to be simple. I assigned the hopping ip (192.168.0.160) to bridge itself (exactly as I did it on linux). Easier said but done. For some reason FreeBSD 12 scripts exclusively allowed only one of two things: either assign ip to the bridge or bring bge0 up. As soon as I fix one, script resets the other. After debuging network scripts I'm still unsure. It maybe just me being ignorant about BSD rc.conf craft or this is a bug:


```
# from /etc/rc.d/netif
        if [ -f /etc/rc.d/bridge -a -n "$cmdifn" ] ; then
                # I had to add quotes below. no way it can pass a list of interfaces otherwise.:(
                /etc/rc.d/bridge start "$cmdifn"
        fi
```

rc.conf if somebody cares:

```
network_interfaces="lo0 bge0 bridge0 ab0"
autobridge_interfaces="ab0"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
autobridge_ab0="bge0"
ifconfig_bge0="up"
ifconfig_ab0="192.168.0.160 netmask 255.255.255.0"
```


----------



## SirDice (Jan 18, 2019)

crypt47 said:


> ```
> network_interfaces="lo0 bge0 bridge0 ab0"
> autobridge_interfaces="ab0"
> autobridge_ab0="bge0"
> ```


Remove those.

And use this:

```
ifconfig_ab0="addm bge0 inet 192.168.0.160 netmask 255.255.255.0"
```


----------



## crypt47 (Jan 18, 2019)

SirDice, this doesn't actually set an ip on bridge. Though adds a member to the bridge.


----------



## crypt47 (Jan 18, 2019)

considering it's a raw ifconfig command it works without "inet", will check later.


----------



## SirDice (Jan 18, 2019)

No, you're right. I was sure I had that configured myself at some point. But I've used a completely different method (I'm using sysutils/vm-bhyve to create/manage them). It's definitely possible to add an IP address to a bridge(4) interface but getting it correct in rc.conf is a bit tricky and doesn't work the way I expected it to work.


----------



## crypt47 (Jan 20, 2019)

In the end there are two sets of directives, that will work:


```
network_interfaces="lo0 bge0 bridge0 ab0"
autobridge_interfaces="ab0"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
autobridge_ab0="bge0"
ifconfig_bge0="up"
ifconfig_ab0="192.168.0.160 netmask 255.255.255.0"
```

and


```
network_interfaces="lo0 bge0 bridge0 ab0"
ifconfig_bge0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="ab0"
ifconfig_ab0="addm bge0 192.168.0.160 netmask 255.255.255.0"
```

In both cases network_interfaces list is mandatory. Quotation fix I used is not, though I probably raise a bug cause it's more like a coincidence.


----------

