# bridge bhyve



## Ofloo (Jan 19, 2018)

I've got 2 bhyve systems running on a tap-to-tap vlan with bridge, .. on FreeBSD 11.1-STABLE. Somehow the throughput from one system to an other is only 600mbit. Both are running FreeBSD 11.1-RELEASE inside bhyve.
Does any one have any suggestions what the reason could be?


```
# ifconfig bridge3
bridge3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: vm-private
   ether 02:8d:f7:ea:7a:03
   nd6 options=1<PERFORMNUD>
   groups: bridge
   id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
   member: tap9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 30 priority 128 path cost 2000000
   member: tap8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 29 priority 128 path cost 2000000
   member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 24 priority 128 path cost 2000000
   member: vlan3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 14 priority 128 path cost 2000000
```

On FreeBSD 11.1-RELEASE host I get 3gbit throughput, however this systems nic drivers are not in RELEASE, .. "Ethernet Connection X553 1GbE", .. so I installed STABLE instead. Not sure if it's STABLE related. However on a tap to tap bridge I wouldn't know where else to look.


----------



## Ofloo (Jan 27, 2018)

Did svn up and recompiled kernel, there seem some improvements, .. which makes me wonder if it's in stable.


```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 59390 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.02 GBytes   872 Mbits/sec
```
From 600mbits to 872mbits


----------



## Ofloo (Feb 1, 2018)

FreeBSD 11.1-STABLE #6 r328591: Wed Jan 31 00:50:02 CET 2018


```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 60254 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.06 GBytes   956 Mbits/sec
```


----------



## aragats (Feb 1, 2018)

Just for statistics (not answering the original question):
I tested the throughput between two bhyve guests and the host on 11.1-RELEASE, however, I don't use vlan.
Windows guest <-> Linux guest: 5Gbit/sec
Windows guest <-> Host: 5.3Gbit/sec
Linux guest <-> Host: 11Gbit/sec


----------



## Ofloo (Feb 1, 2018)

freebsd11.1 release with vlan

```
# iperf -c 10.13.17.200
------------------------------------------------------------
Client connecting to 10.13.17.200, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.17.10 port 15355 connected with 10.13.17.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.88 GBytes  1.61 Gbits/sec
```

freebsd11.1 without vlan

```
# iperf -c brain
------------------------------------------------------------
Client connecting to brain, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.5 port 45728 connected with 192.168.1.9 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.64 GBytes  3.12 Gbits/sec
```

freebsd11.1 epair vimage jail without vlan

```
# iperf -c peanut
------------------------------------------------------------
Client connecting to peanut, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.5 port 43887 connected with 192.168.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.43 GBytes  2.95 Gbits/sec
```

FreeBSD stable performs a lot worse. The above are all freebsd guests

Another thing is why would it be slower cause it has vlan, .. it shouldn't translate it since it's untagged traffic it's only tagged traffic if it's leaving the vlan bridge which it isn't.

EDIT: The cpu load has effect on the test, these last tests of freebsd11.1 will be a bit beter since the that system was compiling at the time. The tests of the FREEBSD11-STABLE has no load.


----------



## Ofloo (Feb 4, 2018)

FreeBSD 11.1-STABLE #7 r328772: Fri Feb  2 17:34:54 CET 2018


```
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 54625 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   724 MBytes   607 Mbits/sec
```


----------



## Ofloo (Feb 4, 2018)

FreeBSD 11.1-STABLE #8 r328837: Sun Feb  4 16:24:55 CET 2018


```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 19678 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.01 GBytes   868 Mbits/sec
```


----------



## Ofloo (Feb 6, 2018)

FreeBSD 11.1-STABLE #9 r328915: Tue Feb  6 12:34:31 CET 2018

```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 37898 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   943 MBytes   791 Mbits/sec
```


----------



## Ofloo (Feb 8, 2018)

FreeBSD 11.1-STABLE #10 r328971: Wed Feb  7 22:17:53 CET 2018


```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 63848 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1010 MBytes   847 Mbits/sec
```


----------



## Ofloo (Feb 9, 2018)

FreeBSD 11.1-STABLE #11 r329023: Fri Feb  9 04:20:32 CET 2018


```
# iperf -c 10.13.37.201
------------------------------------------------------------
Client connecting to 10.13.37.201, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.13.37.202 port 53528 connected with 10.13.37.201 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   888 MBytes   745 Mbits/sec
```


----------



## Eric A. Borisch (Feb 9, 2018)

You might want to perform multiple tests on each version to get some feel for the stability of this measurement; single point measurements under each condition (version #) aren’t especially informative.

ministat(1) is a simple way to evaluate repeated measurements (from each line of a file) and their changes over different conditions (different files.)


----------



## Eric A. Borisch (Feb 9, 2018)

And it goes without saying, and you seem to indicate it above, that you should have no other load on any VMs or the host system when testing if you're trying to tease out differences between versions...

Is this a multi-socket system?


----------



## Ofloo (Feb 11, 2018)

What do you mean by multisocket? I think cpu ? https://www.supermicro.com/products/motherboard/Atom/A2SDi-4C-HLN4F.cfm with 16Gbit ecc ram (kingston)

And no there's nothing really going on on the system. Except the test bhyves of course, no compiles or whatever going on.

I'm not quite sure what it is you want me to do with ministat.


----------



## Eric A. Borisch (Feb 11, 2018)

Well,”single socket” — just like what is listed on the page you linked; one physical CPU. (As opposed to cores.) Just an added complexity.

My suggestion is to run each test setup multiple times. The consistency (or lack thereof) will inform you on how much of a difference you need between measurements to be able to say “this version is actually performing better.”

Ministat can crunch the numbers within and across test setups to tell you how confidently you can say the results are different. You would need to create one file for each test setup, with one BW value per line in each file to use it.

http://www.ivoras.net/blog/tree/2009/Dec-2009-12-02.using-ministat.html


----------



## Eric A. Borisch (Feb 11, 2018)

See also: https://wiki.freebsd.org/BenchmarkAdvice


----------



## Ofloo (Feb 11, 2018)

I'm not quite sure how to use iperf with ministat. To me doesn't make much sense how those 2 programs are supposed to work together. If I read the manual page of ministat, then it's supposed to make some sort of ascii diagram ..? Which it reads from a file? Kinda what rrdtool would do.


----------



## Eric A. Borisch (Feb 11, 2018)

You will need to do some work. You might be able to use the CSV option from iperf, but I haven’t looked at it.

At a minimum, you can manually create a file, perhaps named by your svn version #. After you run iperf five-ten times, (the more, the better) write the BW numbers (one per line) in the appropriate file. When you are done, you would run `ministat svn1 [svn2 ..]`. With multiple result files, it will give you some information about how confident you can be that the results are actually different.

Please review the linked wiki page from my last post. If people are going to be inspired to help track down something, you need to show that it actually exists to track down.


----------



## Ofloo (Feb 12, 2018)

Ok now I get the idea on what it is you want me to do, ..


----------



## Ofloo (Feb 12, 2018)

I hope this is satisfying
One against localhost as a reference maybe ?

```
# ministat -A -s -d , -C 9 r329023-local r329152-local
x r329023-local
+ r329152-local
    N           Min           Max        Median           Avg        Stddev
x  11 7.4889298e+09 8.5249229e+09 8.4903199e+09 8.3869189e+09 3.0079192e+08
+  11 5.8070139e+09 8.4861256e+09 8.4116767e+09 8.1610456e+09 7.8515142e+08
No difference proven at 95.0% confidence
```


One on the bridge with vlan tag

```
# ministat -A -s -d , -C 9 r329023-bridge r329152-bridge
x r329023-bridge
+ r329152-bridge
    N           Min           Max        Median           Avg        Stddev
x  11 7.7175194e+08 9.2484403e+08 8.5266638e+08 8.5269877e+08      40599803
+  11 4.2047898e+08 9.4476698e+08 6.8418129e+08 6.8457582e+08 1.8123296e+08
Difference at 95.0% confidence
        -1.68123e+08 +/- 1.16812e+08
        -19.7166% +/- 13.6991%
        (Student's t, pooled s = 1.31327e+08)
```

To be honest this means less to me, looks very difficult to get a grasp on what is going on.


----------



## Eric A. Borisch (Feb 14, 2018)

OK, a couple of things:

You need to tell ministat what column to run statistics on in this case. Right now, it's running them on the first column (Date and time). `-C 9` looks like the right choice (bytes/sec).

You'll want to remove the last entry (cumulative values) from the text files. There doesn't seem to be a switch to iperf to request omitting the line, unfortunately.

Your 329152-bridge values show a 2:1 range in performance within a test. This indicates that the system isn't stable enough to measure -- something is causing significant variation in your results. Ideally you can identify what this and remove it (top can help identify culprits; both on the host and the guests.). Kill all bhyve instances other than those needed for the test. You want the variation within a test case to be small, such that you can tell the difference between test cases. 

Alternatively, having iperf report after longer time periods would likely "work", but it would be better to identify and remove the underlying variablity than to just average over it.

Once you've got all that done, when you run ministat with (just) two local result files, they will hopefully show minimal difference, with local vs. bhyve, you'll likely see a difference (there is more processing to be done), and then ultimately between two bhyve cases to see if there is a (statistically sound) difference in measurements.


----------



## Ofloo (Feb 14, 2018)

If there's anything running why isn't this showing in the localhost test? This is a newly installed system not much going on.


```
clear_tmp_enable="yes"
defaultrouter="192.168.1.1"
local_unbound_enable="YES"
powerd_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"

# network
cloned_interfaces="lagg0"
ifconfig_ix0="up"
ifconfig_ix1="DHCP"
ifconfig_ix1_ipv6="inet6 accept_rtadv"
ifconfig_ix2="up"
ifconfig_ix3="up"
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_igb2="up"
ifconfig_igb3="up"

# network lacp
#ifconfig_lagg0="laggproto lacp laggport ix2 laggport ix3"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3"

# network vlans
vlans_ix1="10"
ifconfig_ix1_10="DHCP"
ifconfig_ix1_10_ipv6="inet6 accept_rtadv"

# daemons
sshd_enable="YES"
syslogd_flags="-ss"
ntpd_enable="yes"
ntpd_sync_on_start="yes"

# virtualisation
vm_enable="yes"
vm_dir="zfs:zroot/bhyve"
vm_list=""
vm_delay="10"
```


----------



## Ofloo (Feb 14, 2018)

> You'll want to remove the last entry (cumulative values) from the text files. There doesn't seem to be a switch to iperf to request omitting the line, unfortunately.


Are refering to the empty line or the last result?
`iperf -c 10.13.37.201 -t 11 -i 1 -y C | head -n 10>file`


```
# ministat -A -s -d , -C 9 r329023-local r329152-local r329192-loca
x r329023-local
+ r329152-local
* r329192-local
    N           Min           Max        Median           Avg        Stddev
x  11 7.4889298e+09 8.5249229e+09 8.4903199e+09 8.3869189e+09 3.0079192e+08
+  11 5.8070139e+09 8.4861256e+09 8.4116767e+09 8.1610456e+09 7.8515142e+08
No difference proven at 95.0% confidence
*  11 7.2456602e+09 8.5941289e+09 8.5417001e+09 8.4176434e+09 3.9150634e+08
No difference proven at 95.0% confidence
```


```
# ministat -A -s -d , -C 9 r329023-bridge r329152-bridge r329192-bridge
x r329023-bridge
+ r329152-bridge
* r329192-bridge
    N           Min           Max        Median           Avg        Stddev
x  11 7.7175194e+08 9.2484403e+08 8.5266638e+08 8.5269877e+08      40599803
+  11 4.2047898e+08 9.4476698e+08 6.8418129e+08 6.8457582e+08 1.8123296e+08
Difference at 95.0% confidence
        -1.68123e+08 +/- 1.16812e+08
        -19.7166% +/- 13.6991%
        (Student's t, pooled s = 1.31327e+08)
*  11 6.5431142e+08 9.2379546e+08 9.0282394e+08 8.2501253e+08 1.0984533e+08
No difference proven at 95.0% confidence
```


----------



## Eric A. Borisch (Feb 14, 2018)

One quick thing to try would be to turn off powerd when benchmarking. I’m not sure it buys you much (power/heat) on a modern server where the CPU already dynamically throttles cores, but it can certainly introduce variability.


----------



## Ofloo (Feb 14, 2018)

Noticed it too disabled it already, no performance gain though. Shouldn't have been enabled cpu doesn't support cpufreq anyway.


----------



## Ofloo (Feb 18, 2018)

I'm wondering the performance on bhyve should be nearly the same as on the host system or am I missing something? So why is it that the network performance on stable and on release are cut in half?

FreeBSD 11.1-RELEASE-p6

```
# iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 47.9 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 42358 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  55.6 GBytes  47.7 Gbits/sec
```
FreeBSD 11.1-RELEASE-p6 BHYVE

```
# iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 47.9 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 12619 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  23.6 GBytes  20.3 Gbits/sec
```
FreeBSD 11.1-STABLE #14 r329449

```
# iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 47.9 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 17872 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  20.4 GBytes  17.5 Gbits/sec
```
FreeBSD 11.1-STABLE #14 r329449 BHYVE

```
# iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 47.9 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 50990 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.65 GBytes  8.28 Gbits/sec
```

And when this goes over vtnet it goes down roughly by 6-8 and when a vlan tag is added this goes down again by 2


----------

