# Network Issues w/ LAGG + VMs



## LordInateur (Dec 18, 2021)

Hello, all! I have an interesting setup that I'm hoping to get a little bit of help with. Before I start: please yell at me if some of my formatting is wonky-- I'm used to markdown, but not so much the styling that the FreeBSD Forums asks for, so if I've done something that needs to be different, I'm happy to learn. Okay: I have a network setup with a traffic flow that I want to resemble the following, from _outermost_ to _innermost_:

Public-facing WAN on a bare-metal OPNSense instance; the OPNSense instance is also the DHCP server for everything under it (and this has been relatively successful), so while OPNSense serves multiple interfaces, each interface with a subnet, the one we're working with is on 10.0.9.0/24 (and the OPNSense instance / DHCP server is at 10.0.9.1)
Mikrotik SFP+ switch with an uplink to an interface on the OPNSense box marked OPT1; the Mikrotik switch sits at 10.0.9.2
Hypervisor host running FreeBSD 13.0-RELEASE with two physical interfaces dubbed bxe0 and bxe1 respectively, uplinked to the Mikrotik switch with two DAC lines (each individual line supports 10Gbps); neither of these interfaces has an IP assigned to it
Cloned link aggregation interface on the hypervisor host dubbed lagg0 using the loadbalance protocol with members bxe0 and bxe1; the lagg interface doesn't have an IP address assigned to it
Cloned bridge interface on the hypervisor host dubbed bridge0 with members lagg0 and tap1; the bridge interface has 10.0.9.33 successfully assigned to it through a static DHCP lease with a static ARP entry
Automatically-generated tap1 interface on the hypervisor host, generated by CBSD on behalf of a bhyve.
Guest-side virtual interface dubbed enp0s5
I put the entire hardware workflow there because I wonder if the answer to my question has to do with ARP-based requests traveling through multiple interfaces.

Okay. So, I do have a custom-built kernel for IPFW (yes I know I can just load it at runtime, I chose not to for reasons; I'm not married to those reasons). Additions to the kernel are as follows:


```
#
# IPFWKERNEL -- Generic kernel configuration file for FreeBSD/amd64 w/ IPFW support
# This needs to be located at /usr/src/sys/amd64/IPFWKERNEL
#

include GENERIC
ident IPFWKERNEL

# IPFW
options          IPFIREWALL                  # required for IPFW
options          IPFIREWALL_VERBOSE          # optional; logging
options          IPFIREWALL_VERBOSE_LIMIT    # optional; don't get too many log entries
options          IPDIVERT                    # needed for natd
```

In its current state of debugging, I've disabled some of the NAT configuration options that CBSD provides. Anyhow, here's my boot loader configuration:


```
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
if_lagg_load="YES"
kern.racct.enable=1
vmm_load="YES"
```

And for good measure, here's my /etc/rc.conf:


```
hostname="foo.bar.baz.com"
cloned_interfaces="lagg0 bridge0"
ifconfig_bridge0="addm lagg0 SYNCDHCP"
ifconfig_lagg0="laggproto loadbalance laggport bxe0 laggport bxe1"
ifconfig_bxe0="up"
ifconfig_bxe1="up"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"
cbsd_workdir="/usr/jails"
cbsdrsyncd_enable="YES"
cbsdrsyncd_flags="--config=/usr/jails/etc/rsyncd.conf"
cbsdd_enable="YES"
rcshutdown_timeout="900"
kld_list="vmm if_tuntap if_bridge nmdm"
```

I do have an IPFW ruleset at /etc/ipfw.rules


```
IPF="ipfw -q add"
ipfw -q -f flush

# Loopback 
$IPF 10 allow all from any to any via lo0
$IPF 20 deny all from any to 127.0.0.0/8
$IPF 30 deny all from 127.0.0.0/8 to any
$IPF 40 deny tcp from any to any frag

# Stateful
$IPF 20010 check-state
$IPF 20020 allow tcp from any to any established
$IPF 20030 allow all from any to any out keep-state
$IPF 20040 allow icmp from any to any

# System Services
$IPF 20010 allow tcp from any to any dst-port 53 keep-state # DNS TCP
$IPF 20011 allow tcp from any to any dst-port 53 keep-state # DNS UDP
$IPF 20020 allow udp from any to any 67 in # DHCP SERVER IN
$IPF 20021 allow udp from any to any 67 out # DHCP SERVER OUT
$IPF 20022 allow udp from any to any 68 in # DHCP CLIENT IN
$IPF 20023 allow udp from any to any 68 out # DHCP CLIENT OUT

# User Services
$IPF 20100 allow tcp from any to any 2212 in  # SSH ALT IN
$IPF 20101 allow tcp from any to any 2212 out # SSH ALT OUT

# Catch-All: Deny + Log
$IPF 65534 deny log all from any to any in
```

That being said, when I kick a Bhyve on (in this case, balance1a) I do get a modified ruleset because CBSD adds some rules. `ipfw list` thus yields:


```
00010 allow ip from any to any via lo0
00020 deny ip from any to 127.0.0.0/8
00030 deny ip from 127.0.0.0/8 to any
00040 deny tcp from any to any frag offset
00099 count ip from any to any out via tap1 // Setup by CBSD bhyve start: balance1a
00100 count ip from any to any in via tap1 // Setup by CBSD bhyve start: balance1a
20010 check-state :default
20010 allow tcp from any to any 53 keep-state :default
20011 allow tcp from any to any 53 keep-state :default
20020 allow tcp from any to any established
20020 allow udp from any to any 67 in
20021 allow udp from any to any 67 out
20022 allow udp from any to any 68 in
20023 allow udp from any to any 68 out
20030 allow ip from any to any out keep-state :default
20040 allow icmp from any to any
20100 allow tcp from any to any 2212 in
20101 allow tcp from any to any 2212 out
65534 deny log logamount 1 ip from any to any in
65535 deny ip from any to any
```

(And yeah, I probably do need to fix some of the funkiness around rule 100 just for the sake of clarity.)

Okay, for CBSD, here's some settings I've transcoded from `cbsd initenv-tui`

rcconf = `CBSD Enabled: YES`
nodename = `foo.bar.baz.com`
nodeip = `10.0.9.33` # note: the hypervisor host got this from DHCP, but CBSD got this from manual entry
nodeip6 = `0`
nodedescr = `0`
jnameserver = `10.0.9.1`
nodeippool = `10.0.9.0/24`
fbsdrepo = `1`
repo = `https://bsdstore.ru`
ipfw_enable = `1`
zfsfeat = `1`
jail_interface = `0`
parallel = `5`
stable = `0`
sqlreplica = `1`
statsd_bhyve_enable = `1`
statsd_jail_enable = `1`
statsd_hoster_enable = `1`
natcfg = `0 (not configured)`
And for the CBSD config for the Bhyve in particular (after running `cbsd bconfig jname=balance1a` -- I've skipped some output because I can't imagine that it'd have anything to do with this)

ip4_addr = `10.0.9.129` # this is manually entered for CBSD, but it's also a static lease on the DHCP server
bhyvenic > nic1 = `vtnet` > nic_parent = `bridge0`
Okie dokie, so finally, the problem: I can't seem to get a DHCPOFFER sent to the guest Bhyve. The guest OS is running Ubuntu 20.04 LTS, and here are the contents of /etc/netplan/00-installer-config.yaml:


```
network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s5:
      dhcp4: yes
```

The second entry for `ip addr` reads as follows (I've included the enp0s5 entry but have excluded the lo entry):


```
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_code1 state UP group default qlen 1000
    link/ether 00:a0:98:38:12:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2a0:98ff:fe38:12af/64 scope link
       valid_lft forever preferred_lft forever
```

Okay, I'm not particularly interested in IPv6 here. I do know for certain that the DHCPDISCOVER is making it out to the DHCP server, according to my OPNSense logs. I also do have logs indicating that the DHCP server is responding with a DHCPOFFER, and the MAC addresses matches the one in the code block I noted above.

I will note that I do have similar setups using similar hardware-- I do have a bhyve running in that same datacenter on a hypervisor pointed to an unmanaged switch, which is in turn uplinked to the same OPNSense box (just on a different interface). I also did (but don't currently) have boxes on the Mikrotik switch that did have bhyve instances on them-- but their physical interfaces weren't lagg'd. I've compared some of the above files between the box that's giving me trouble and the boxes that are working, and have tried to make them as similar as possible-- and this is as close as I've gotten at this point.

I've done some generic testing i.e. disabling the firewall to see if it's blocking things, trying to statically set the IP in the guest, etc, and I've not had luck. I'm happy to provide more information if it's needed, and I'm happy for any insight that y'all can provide. Thanks!


----------



## Ole (Dec 18, 2021)

What does the full 'ifconfig' output look like for the host system when the virtual machines are running? Have you tried doing 'ifconfig bridge0 up' + 'ifconfig lagg0 up' ?


----------



## LordInateur (Dec 18, 2021)

Ole said:


> What does the full 'ifconfig' output look like for the host system when the virtual machines are running? Have you tried doing 'ifconfig bridge0 up' + 'ifconfig lagg0 up' ?



`ifconfig` yields the following:


```
bxe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bxe1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    hwaddr 00:0e:1e:9b:dd:52
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    laggproto loadbalance lagghash l2,l3,l4
    laggport: bxe0 flags=4<ACTIVE>
    laggport: bxe1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:e6
    inet 10.0.9.33 netmask 0xffffff00 broadcast 10.0.9.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 1000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
tap1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: balance1a-nic0
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:f8
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 5252
```

Executing `ifconfig bridge0 up` and `ifconfig lagg0 up` appeared to have no effect.


----------



## Ole (Dec 18, 2021)

LordInateur said:


> `ifconfig` yields the following:
> 
> 
> ```
> ...



As far as I can see, the `tap` interface has not been added to the `bridge0` group. Will the interface be added manually:  
	
	



```
ifconfig bridge0 addm tap1
```
 ?


----------



## LordInateur (Dec 18, 2021)

Ole said:


> As far as I can see, the `tap` interface has not been added to the `bridge0` group. Will the interface be added manually:
> 
> 
> 
> ...



Ope, that's my fault. I was testing a theory after my original post and failed to completely revert the changes. The full output of `ifconfig` reads as follows:


```
bxe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bxe1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    hwaddr 00:0e:1e:9b:dd:52
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    laggproto loadbalance lagghash l2,l3,l4
    laggport: bxe0 flags=4<ACTIVE>
    laggport: bxe1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:e6
    inet 10.0.9.33 netmask 0xffffff00 broadcast 10.0.9.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 1000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: balance1a-nic0
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:f8
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 5182
```

(The original issue persists.)


----------



## Ole (Dec 19, 2021)

And where is the Linux guest? (ether: *00:a0:98:38:12:af*).  As far as I understand, OPNSense == *balance1a* VM ?


----------



## LordInateur (Dec 19, 2021)

Ole said:


> And where is the Linux guest? (ether: *00:a0:98:38:12:af*).  As far as I understand, OPNSense == *balance1a* VM ?



No, balance1a is a guest VM on the hypervisor host, which from this point on I'll call hyp01 (though in my original post, I've noted its hostname as foo.bar.baz.com). So, hyp01 is a bare metal machine that physically sits in a rack in some datacenter, and OPNSense sits on another physical machine that I'll call fw01. The MAC address you mention is the one generated and assigned to the virtual interface as presented _within_ the Ubuntu VM, which I've referred to as balance1a but probably ought to have referred to as guest01 to remove that ambiguity. Then, hyp01 and fw01 are physically connected through a Mikrotik switch. fw01 (with OPNSense) serves as the gateway and DHCP server.


----------



## Ole (Dec 19, 2021)

Got it. Why I'm asking - there is a issue in the FreeBSD vtnet (virtio) driver and OPNSense guest may be affected. As dirty work-around: you need to manually switch the interface in the OPNsense guest to *PROMISC* mode:


```
/sbin/ifconfig vtnet0 promisc
```

If this does not help, tcpdump might be a good tool here.


----------



## LordInateur (Dec 19, 2021)

Ole said:


> Got it. Why I'm asking - there is a issue in the FreeBSD vtnet (virtio) driver and OPNSense guest may be affected. As dirty work-around: you need to manually switch the interface in the OPNsense guest to *PROMISC* mode:
> 
> 
> ```
> ...


I'm a bit confused -- in my setup, there is no OPNsense guest (there's a bare metal install of it on a completely separate physical device upstream), and the only guest VM is a Bhyve running Ubuntu 20.04 LTS. The hypervisor host containing the Bhyve is running FreeBSD 13.0-RELEASE. Which of these instances are you talking about?


----------



## Ole (Dec 19, 2021)

LordInateur said:


> I'm a bit confused -- in my setup, there is no OPNsense guest (there's a bare metal install of it on a completely separate physical device upstream), and the only guest VM is a Bhyve running Ubuntu 20.04 LTS. The hypervisor host containing the Bhyve is running FreeBSD 13.0-RELEASE. Which of these instances are you talking about?


I'm sorry, I got confused in your architecture and began to think that theOPNSense is in another virtual machine.


----------



## LordInateur (Dec 19, 2021)

Ole said:


> I'm sorry, I got confused in your architecture and began to think that theOPNSense is in another virtual machine.



Ah yeah, no worries. Maybe this will help to visualize what I'm trying to do (apologies in advance to those who hate images in forum posts):






The issue I've spoken of above specifically regards the failure of enp0s5 on balance1a (the Ubuntu 20.04 LTS bhyve guest instance) to obtain its IP address (10.0.9.129) via DHCP.


----------



## LordInateur (Dec 22, 2021)

Alrighty then, an update for anyone interested. With some help from the CBSD Telegram channel, we've narrowed down the issue to an issue with potentially the lagg0 interface itself. A couple of things we've discovered:

Turns out that it's not DHCP traffic alone-- it's any traffic. No traffic reaches the guest VM in the current configuration.
When an interface is removed from lagg0, the guest VM is able to send and receive traffic.
So, I've tried this with both the loadbalance and roundrobin protocols to no avail. I am trying to avoid LACP, because we're quite certain that the particular Mikrotik switch I have would throttle quite a bit of bandwidth if it had to process that itself. Still working on a fix, and when I have it I'll post it here so that my future self can find it again.


----------



## LordInateur (Dec 23, 2021)

Ok, bumping this thread in an attempt to hopefully get some help from the community on this. I'm attempting to try to solve this by adding a VLAN interface into the mix but I'm sort of just stumbling around for answers at this point. Here's my current /etc/rc.conf --


```
hostname="foo.bar.baz.tld"
cloned_interfaces="lagg0 bridge0 vlan1"
ifconfig_bxe0="up"
ifconfig_bxe1="up"
ifconfig_lagg0="laggproto loadbalance laggport bxe0 laggport bxe1 SYNCDHCP"
ifconfig_vlan1="vlandev lagg0 vlan 1 up"
ifconfig_bridge0="addm vlan1 up"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"
cbsd_workdir="/usr/jails"
cbsdrsyncd_enable="YES"
cbsdrsyncd_flags="--config=/usr/jails/etc/rsyncd.conf"
cbsdd_enable="YES"
rcshutdown_timeout="900"
kld_list="vmm if_tuntap if_bridge nmdm"
```

And my current set of interfaces:


```
bxe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bxe1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    hwaddr 00:0e:1e:9b:dd:52
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:0e:1e:9b:dd:50
    inet 10.0.9.103 netmask 0xffffff00 broadcast 10.0.9.255
    laggproto loadbalance lagghash l2,l3,l4
    laggport: bxe0 flags=4<ACTIVE>
    laggport: bxe1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:e6
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=401<RXCSUM,LRO>
    ether 00:0e:1e:9b:dd:50
    groups: vlan
    vlan: 1 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: balance1a-nic0
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:f8
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 6043
```

So I _think _that I'm looking for is an untagged VLAN implementation. But that's a bit of a new networking area for me and I didn't find much assistance for my current setup in the Handbook, unfortunately. At the end of the day, I'm really looking for:

non-LACP link aggregation (preferably using the loadbalance protocol, for reasons noted above)
access to the hypervisor box itself (so, the physical box needs to have an IP obtained via DHCP)
access to the guest VMs (so, guests need to have IPs obtained via DHCP)


----------



## LordInateur (Dec 24, 2021)

Ok, first off, I've changed the title of this thread from "DHCP Issues w/ Bhyve + CBSD + LAGG" to "Network Issues w/ LAGG + VMs" since I've eliminated some of the possible causes... and clearly this issue isn't a DHCP problem directly, and I don't think CBSD has anything to do with it at this point.

Funny thing I've noticed: I'm able to get network activity on the VM in question if I restart the VM and execute the following (interface order is ambiguous):

`ifconfig bxe1 down`
`ifconfig bxe1 up`
`ifconfig bxe0 down`
`ifconfig bxe0 up`
I have bxe0 and bxe1 as physical interfaces on lagg0 (I've reverted changes regarding VLANs, so I have no VLAN interface at this time). The VM is able to conduct network transactions the moment I turn a single interface off (leaving one up), but the moment I turn the interface back on I lose network connectivity again. However, if I start and stop _both_ interfaces (not at the same time, so I don't lose management access), then I can turn them both back on and the VM can continue on the network.

A really hack-and-slash solve for this would be to introduce a hook that performed this action when the VM restarts. But this is something I really want to avoid-- it seems fundamentally nasty. However, I now know that it's at least _possible_ to have the VMs directly on lagg0 instead of on a VLAN.

Insight on this from the community is still very much welcome.


----------



## LordInateur (Dec 25, 2021)

SirDice would you weigh in on this please? I don't seem to be getting anywhere, but I've seen multiple posts on the forums adjacent to this particular topic, have seen your replies to those messages and therefore know that you've set up stuff like this in the past-- I would very much appreciate your insight. Some questions I have outstanding:

Can bhyves network with a link aggregation interface properly without the use of VLANs? (I'm assuming yes from the above, but I haven't seen anything around to help with that)
If so, why might I be experiencing issues like the above?
If not, what is the proper way to go about setting up a (presumably untagged/native) VLAN such that both the host machine and guest bhyve obtain their IP via DHCP lease?
Thank you.

(I will note that I have since moved back to the GENERIC kernel and have attempted this with IPFW set to open, to no avail.)


----------



## LordInateur (Dec 25, 2021)

Small update for my non-existent crowd: the failover protocol definitely works without a VLAN. So, I think this is possibly a frame issue with the loadbalance and roundrobin protocols. Thinking I'm getting close to knocking this one out. The saga continues.


----------



## LordInateur (Dec 26, 2021)

*sigh* LACP definitely works, so my setup is working-- I think I'll just have to see if it's truly limited by the Mikrotik's hardware. But, as I was really gunning for loadbalance to work here, and since that particular piece was never resolved, I won't be marking this as "Solved" for now.


----------

