# Avahi/mDNS not working in shared IP jails



## floogs (Aug 3, 2019)

I've been running six VNET-enabled jails via iocage that I wanted to revert to shared IP, as I've had some system crashes that I suspect are related to VNET issues. (The iocage documentation describes VNET as experimental and prone to causing instability.)

After making the switch, the three jails with services that rely on avahi/multicast traffic for network announce and discovery of clients -- CUPS, Homebridge, and forked-daapd -- have stopped working, while the other jails are still working fine, which leads me to think something about the default shared IP jail config prevents avahi from working right. I've found a number of threads on this forum going back a solid decade (!!) asking about this issue but no one seems to have arrived at a definitive cause or solution.

Does anyone have any ideas? I'm relatively inexperienced with FreeBSD and even less so high-level networking, but figured my network config is the best place to start. Also tried enabling allow_raw_sockets on the affected jails with no change.

Host system with shared IP jails (the first IP ending .52 is the host, the others are jails):


```
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 1c:b7:2c:af:91:35
    hwaddr 1c:b7:2c:af:91:35
    inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.98 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.99 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.92 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.175 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.219 netmask 0xffffff00 broadcast 192.168.1.255
    inet 192.168.1.133 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:89:50:69:d9:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
```

Sample shared IP jail:


```
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 1c:b7:2c:af:91:35
    hwaddr 1c:b7:2c:af:91:35
    inet 192.168.1.133 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:89:50:69:d9:00
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
```

Sample VNET jail:


```
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 1c:b7:2c:bd:67:e6
    hwaddr 02:d2:d0:00:05:0b
    inet 192.168.1.219 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair
```


----------



## floogs (Aug 4, 2019)

Well, I spoke too soon -- even the jails that don't rely on avahi/mDNS only worked normally for a short time after I changed them over to shared IP. None of my Plex devices can see the Plex server, and my Unifi access point controller won't load in a browser. Additionally, I have all the hostnames on my LAN mapped to their respective IPs, and none of those will resolve for the shared IP jails anymore either. All of this works fine if I switch back to VNET, just with the potential for system crashes.

So I guess more generally: what is it about the shared IP configuration that makes the jails' network access so much more limited and is there a way to expose all the same features as in a VNET scenario to make all this work again?


----------



## SirDice (Aug 5, 2019)

Don't assign the jail addresses to em0. Don't assign an IP address to em0 at all. Mixing an IP address on a physical interface (em0) and attaching a bridge(4) doesn't work like you think it does. The bridge(4) interface hooks into the network stack much higher than you expect causing strange results.


----------



## floogs (Aug 5, 2019)

Thanks for the response. iocage does expect you to give an interface to attach to when you specify the IP for the jail, so I guess I misinterpreted the documentation and set each one to the physical adapter (as I said I'm pretty inexperienced with this sort of networking).

Which interface would be the appropriate one to assign the jail IPs to? Is a bridge for each jail the right way to go? If so, do I need to create a separate bridge for each jail or one common one among all of them?

If there's a more general guide to networking you'd recommend reading so I can get a better general understanding of how this works, rather than answering a bunch of specific questions, please let me know.


----------



## abishai (Aug 5, 2019)

SirDice said:


> Don't assign the jail addresses to em0


Why not? If jails have ip from the very subnet as em0, it's perfect solution. Attach em0 with assigned IP to bridge is not a criminal as well. It will work. Jails without network stack wont receive multicasts, so zero configuration and discovery will not work.
floogs Are you using pf firewall inside your VNET jails ? If you still on 11 branch, I know the patch that will fix panics.


----------



## SirDice (Aug 6, 2019)

abishai said:


> Why not?


Because of the bridge(4).



abishai said:


> If jails have ip from the very subnet as em0, it's perfect solution.


It is. But not if it's tied to a bridge(4). Like I said, there's something counter-intuitive here.

Ran into a similar issue when I had my management IP attached to igb0 (in my case) and a bunch of VMs tied to a bridge(4) which was also tied to igb0. Fairly straight-forward, or so I thought. It worked better when I removed the IP address from igb0 and moved it to the bridge(4). But I still had weird connectivity issues. In the end I designated a _different_ interface (igb1) as the management interface and removed any and all IP settings from igb0 and the bridge.


----------

