# Backport of if_bridge patch for better performance to 12.1



## zirias@ (May 18, 2020)

Hi all,

here's a recent commit to head (-CURRENT): https://svnweb.freebsd.org/base?view=revision&revision=360345

I thought it might be worthwile to test that with 12.1, so I backported the patch. So far, I only did a quick smoke test with two VNET jails using epair devices connected by a bridge. Using dd and nc, I managed to reach a data rate of 240 MB/s on my old desktop PC, so that doesn't look like an improvement, but at least it's a smoke test that the bridge still works  Maybe the epair devices or my aging CPU are limiting performance, I don't know…

So, *if you feel advanturous* enough to trust my backporting work, please try the patch attached here, *maybe* it improves some setups  Of course, no guarantees I didn't break anything.


----------



## sahlawat (May 24, 2020)

A 4x network performance patch, I had to try this; really wanted this to work, but ...

1. The patch applies clean and compiles with 12.1-p5 r361429M
2. No system crashes - I have 4 bridges with 5 VLANs and a 2 interface lagg port connecting 45 jails with VNET interfaces.

Performed 2 simple iperf3 tests:
1. Jail (hub) to Jail (.123)
2. External pfsense (fw) to Jail (.123)
Jail 192.168.0.123 was running the iperf3 server in both cases


Performance Before Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[  5] local 192.168.0.50 port 43309 connected to 192.168.0.123 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   374 MBytes  3.13 Gbits/sec    0   2.00 MBytes
[  5]   1.00-2.00   sec   384 MBytes  3.22 Gbits/sec    0   2.00 MBytes
[  5]   2.00-3.00   sec   382 MBytes  3.20 Gbits/sec    0   2.00 MBytes
[  5]   3.00-4.00   sec   382 MBytes  3.20 Gbits/sec    0   2.00 MBytes
[  5]   4.00-5.00   sec   385 MBytes  3.23 Gbits/sec    0   2.00 MBytes
[  5]   5.00-6.00   sec   386 MBytes  3.24 Gbits/sec    0   2.00 MBytes
[  5]   6.00-7.00   sec   384 MBytes  3.22 Gbits/sec    0   2.00 MBytes
[  5]   7.00-8.00   sec   389 MBytes  3.26 Gbits/sec    0   2.00 MBytes
[  5]   8.00-9.00   sec   387 MBytes  3.25 Gbits/sec    0   2.00 MBytes
[  5]   9.00-10.00  sec   387 MBytes  3.25 Gbits/sec    0   2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.75 GBytes  3.22 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.75 GBytes  3.22 Gbits/sec                  receiver
iperf Done.

[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[  5] local 192.168.0.5 port 41841 connected to 192.168.0.123 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   1.00-2.00   sec   108 MBytes   903 Mbits/sec    0   65.1 KBytes
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0   65.1 KBytes
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec    0   65.1 KBytes
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec    0   65.1 KBytes
[  5]   5.00-6.00   sec   110 MBytes   919 Mbits/sec    0   65.1 KBytes
[  5]   6.00-7.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec    0   65.1 KBytes
[  5]   8.00-9.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec    0   65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   932 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.09 GBytes   932 Mbits/sec                  receiver
iperf Done.


Performance After Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[  5] local 192.168.0.50 port 44597 connected to 192.168.0.123 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  29.3 MBytes   246 Mbits/sec    0   64.9 KBytes
[  5]   1.00-2.00   sec  26.1 MBytes   219 Mbits/sec    0   64.9 KBytes
[  5]   2.00-3.00   sec  32.7 MBytes   274 Mbits/sec    0   64.9 KBytes
[  5]   3.00-4.00   sec  35.6 MBytes   298 Mbits/sec    0   64.9 KBytes
[  5]   4.00-5.00   sec  28.3 MBytes   238 Mbits/sec    0   64.9 KBytes
[  5]   5.00-6.00   sec  35.2 MBytes   296 Mbits/sec    0   64.9 KBytes
[  5]   6.00-7.00   sec  28.6 MBytes   240 Mbits/sec    0   64.9 KBytes
[  5]   7.00-8.00   sec  32.4 MBytes   272 Mbits/sec    0   64.9 KBytes
[  5]   8.00-9.00   sec  25.5 MBytes   214 Mbits/sec    0   64.9 KBytes
[  5]   9.00-10.00  sec  36.2 MBytes   304 Mbits/sec    0   64.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   310 MBytes   260 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   310 MBytes   260 Mbits/sec                  receiver
iperf Done.

[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[  5] local 192.168.0.5 port 16638 connected to 192.168.0.123 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   937 Mbits/sec    0   65.1 KBytes
[  5]   1.00-2.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   3.00-4.00   sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
[  5]   4.00-5.00   sec   109 MBytes   917 Mbits/sec    0   65.1 KBytes
[  5]   5.00-6.00   sec   112 MBytes   938 Mbits/sec    0   65.1 KBytes
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec    0   65.1 KBytes
[  5]   7.00-8.00   sec   112 MBytes   937 Mbits/sec    0   65.1 KBytes
[  5]   8.00-9.00   sec   112 MBytes   938 Mbits/sec    0   65.1 KBytes
[  5]   9.00-10.00  sec   112 MBytes   939 Mbits/sec    0   65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.09 GBytes   935 Mbits/sec                  receiver
iperf Done.

Given the inter-jail throughput drop I restored the 12.1 bridge files.

Next - will standup another server with 13-current and see what performance I get on that branch.


----------



## pos (May 28, 2020)

I have read of Krystofs great bridge work, and I welcome it *very* much.

The existing bridge code is not super. But works. I do not understand why you see so bad numbers... I have a virtual linux on top of FreeBSD 12.0. The virtual CentOS 7 server is connected over a FreeBSD bridge on Bhyve host. That virtual server gets 7.2 Gbit/s (over 7 routers hops to a target on the internet from my location). Speed test site is "bredbandskollen.se" with CLI client from http://www.bredbandskollen.se/om/mer-om-bbk/bredbandskollen-cli/ (also on git hub). Yes, I have a 10 Gbit internet connection. If not using a FreeBSD bridge I get 8.5 to 9 Gbit. Using a bridge in linux KVM in the same way I do not se any performance drop at all over the bridge. So if the systems are correctly configured the FreeBSD bridges do give half decent performance (but eat resources if you push much traffic).

So my personal opinion is to wait and upgrade to FreeBSD 13 if it is included there. But of course, if you have time available. Why not


----------



## sahlawat (Jun 11, 2020)

Took a bit longer than expected, had to upgrade my testbed to 25G to make sure I was not hitting any network bandwidth limit.

Unfortunately this specific testing shows degraded base network performance in 13-current, so I can't really do any useful bridge performance comparisons across 12.1-release and 13-current. Will revisit this once we have 13.0 RC builds.


Previously (post above) I had applied the back-ported patch to 12.1 and done "jail to jail on same server" testing on my primary server.
FreeBSD 12.1 vanilla:

```
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec receiver
```

FreeBSD 12.1 w/ back-ported patch:

```
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec receiver
```
The "jail to external server" testing was throttled by the 1G network bandwidth to the external server, so did not provide any insights. Given the jail to jail throughput drop of the back-ported patch I am back to using the 12.1 vanilla bridge files.


13-current Vs 12.1-release - simple iperf test

#on both lab17 & lab18:

```
ifconfig bridge1 create
ifconfig bridge1 addm bge0 up
ifconfig bridge2 create
ifconfig bridge2 addm bge1 up
ifconfig bridge11 create
ifconfig bridge11 addm bnxt0 up
ifconfig bridge12 create
#ifconfig bridge12 addm bnxt1 up
#ifconfig bridge12 deletem bnxt1
```

#on lab17:

```
ifconfig bnxt0 10.0.11.17/24 up
ifconfig bnxt1 10.0.12.17/24 up
```

#on lab18:

```
ifconfig bnxt0 10.0.11.18/24 up
ifconfig bnxt1 10.0.12.18/24 up
```

lab17 & lab18 servers are connected over 25G SFP28 links via a switch.
bnxt0 interfaces are in VLAN 11 and bnxt1 interfaces are in VLAN 12



```
root@lab17:~ # uname -a
FreeBSD lab17 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r361567: Thu May 28 04:59:08 UTC 2020     [email]root@releng1.nyi.freebsd.org[/email]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
root@lab17:~ #

root@lab18:~ # uname -a
FreeBSD lab18 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r361567: Thu May 28 04:59:08 UTC 2020     [email]root@releng1.nyi.freebsd.org[/email]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
root@lab18:~ #[/codfe]


Bridged Interfaces: server to server
[code]
Accepted connection from 10.0.11.18, port 57363
[  5] local 10.0.11.17 port 5201 connected to 10.0.11.18 port 20451
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   152 MBytes  1.28 Gbits/sec
[  5]   1.00-2.00   sec   258 MBytes  2.16 Gbits/sec
[  5]   2.00-3.00   sec   262 MBytes  2.20 Gbits/sec
[  5]   3.00-4.00   sec   270 MBytes  2.27 Gbits/sec
[  5]   4.00-5.00   sec   278 MBytes  2.33 Gbits/sec
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec
[  5]   8.00-9.00   sec   252 MBytes  2.12 Gbits/sec
[  5]   9.00-10.00  sec   265 MBytes  2.22 Gbits/sec
[  5]  10.00-10.00  sec   113 KBytes  2.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.51 GBytes  2.16 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.11.17
Connecting to host 10.0.11.17, port 5201
[  5] local 10.0.11.18 port 20451 connected to 10.0.11.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   153 MBytes  1.28 Gbits/sec    7    218 KBytes
[  5]   1.00-2.00   sec   258 MBytes  2.16 Gbits/sec   15    299 KBytes
[  5]   2.00-3.00   sec   262 MBytes  2.20 Gbits/sec   12    242 KBytes
[  5]   3.00-4.00   sec   270 MBytes  2.27 Gbits/sec   10    229 KBytes
[  5]   4.00-5.00   sec   277 MBytes  2.33 Gbits/sec   13    252 KBytes
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec   15    286 KBytes
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec   12    321 KBytes
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec   11    336 KBytes
[  5]   8.00-9.00   sec   253 MBytes  2.12 Gbits/sec  135    348 KBytes
[  5]   9.00-10.00  sec   265 MBytes  2.22 Gbits/sec   13    308 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.52 GBytes  2.16 Gbits/sec  243             sender
[  5]   0.00-10.00  sec  2.51 GBytes  2.16 Gbits/sec                  receiver

iperf Done.[/code


Direct Interfaces: server to server
[code]
Accepted connection from 10.0.12.18, port 24137
[  5] local 10.0.12.17 port 5201 connected to 10.0.12.18 port 19922
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   403 MBytes  3.38 Gbits/sec
[  5]   1.00-2.00   sec   757 MBytes  6.35 Gbits/sec
[  5]   2.00-3.00   sec   758 MBytes  6.35 Gbits/sec
[  5]   3.00-4.00   sec   760 MBytes  6.38 Gbits/sec
[  5]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   5.00-6.00   sec   760 MBytes  6.37 Gbits/sec
[  5]   6.00-7.00   sec   758 MBytes  6.36 Gbits/sec
[  5]   7.00-8.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   8.00-9.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.12.17
Connecting to host 10.0.12.17, port 5201
[  5] local 10.0.12.18 port 19922 connected to 10.0.12.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   404 MBytes  3.39 Gbits/sec   14    268 KBytes
[  5]   1.00-2.00   sec   757 MBytes  6.35 Gbits/sec    2    354 KBytes
[  5]   2.00-3.00   sec   758 MBytes  6.35 Gbits/sec    6    259 KBytes
[  5]   3.00-4.00   sec   760 MBytes  6.38 Gbits/sec    4    349 KBytes
[  5]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec    6    254 KBytes
[  5]   5.00-6.00   sec   760 MBytes  6.37 Gbits/sec    4    344 KBytes
[  5]   6.00-7.00   sec   758 MBytes  6.36 Gbits/sec    6    247 KBytes
[  5]   7.00-8.00   sec   759 MBytes  6.37 Gbits/sec    4    338 KBytes
[  5]   8.00-9.00   sec   759 MBytes  6.37 Gbits/sec   15    269 KBytes
[  5]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec    4    356 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec   65             sender
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec                  receiver

iperf Done.
```


jail to jail on same server

```
Accepted connection from 10.0.11.117, port 43039
[  5] local 10.0.11.127 port 5201 connected to 10.0.11.117 port 55213
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   1.00-2.00   sec   182 MBytes  1.52 Gbits/sec
[  5]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   184 MBytes  1.54 Gbits/sec
[  5]   5.00-6.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   6.00-7.00   sec   154 MBytes  1.29 Gbits/sec
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec
[  5]   8.00-9.00   sec   136 MBytes  1.14 Gbits/sec
[  5]   9.00-10.00  sec   137 MBytes  1.15 Gbits/sec
[  5]  10.00-10.00  sec   771 KBytes  1.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec                  receiver
-----------------------------------------------------------

test-117 ~ # iperf3 -c 10.0.11.127
Connecting to host 10.0.11.127, port 5201
[  5] local 10.0.11.117 port 55213 connected to 10.0.11.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   175 MBytes  1.46 Gbits/sec    0   1.61 MBytes
[  5]   1.00-2.00   sec   182 MBytes  1.52 Gbits/sec    0   1.61 MBytes
[  5]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0   1.61 MBytes
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec    0   1.61 MBytes
[  5]   4.00-5.00   sec   184 MBytes  1.55 Gbits/sec    0   1.61 MBytes
[  5]   5.00-6.00   sec   181 MBytes  1.52 Gbits/sec    0   1.61 MBytes
[  5]   6.00-7.00   sec   153 MBytes  1.29 Gbits/sec    0   1.61 MBytes
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec   136 MBytes  1.14 Gbits/sec    0   1.61 MBytes
[  5]   9.00-10.00  sec   137 MBytes  1.15 Gbits/sec    0   1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec                  receiver

iperf Done.
```


jail to jail on different servers

```
Accepted connection from 10.0.12.128, port 56410
[  5] local 10.0.12.127 port 5201 connected to 10.0.12.128 port 61265
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   126 MBytes  1.06 Gbits/sec
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
[  5]   2.00-3.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   3.00-4.00   sec   156 MBytes  1.31 Gbits/sec
[  5]   4.00-5.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   5.00-6.00   sec   175 MBytes  1.46 Gbits/sec
[  5]   6.00-7.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   7.00-8.00   sec   182 MBytes  1.53 Gbits/sec
[  5]   8.00-9.00   sec   195 MBytes  1.64 Gbits/sec
[  5]   9.00-10.00  sec   188 MBytes  1.58 Gbits/sec
[  5]  10.00-10.02  sec  2.47 MBytes  1.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec  1.69 GBytes  1.45 Gbits/sec                  receiver
-----------------------------------------------------------

test-128 ~ # iperf3 -c 10.0.12.127
Connecting to host 10.0.12.127, port 5201
[  5] local 10.0.12.128 port 61265 connected to 10.0.12.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec    0   1.61 MBytes
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec    0   1.61 MBytes
[  5]   2.00-3.00   sec   173 MBytes  1.45 Gbits/sec    0   1.61 MBytes
[  5]   3.00-4.00   sec   155 MBytes  1.30 Gbits/sec    0   1.61 MBytes
[  5]   4.00-5.00   sec   171 MBytes  1.43 Gbits/sec    0   1.61 MBytes
[  5]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.61 MBytes
[  5]   6.00-7.00   sec   173 MBytes  1.45 Gbits/sec    0   1.61 MBytes
[  5]   7.00-8.00   sec   183 MBytes  1.54 Gbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec   195 MBytes  1.63 Gbits/sec    0   1.61 MBytes
[  5]   9.00-10.00  sec   188 MBytes  1.58 Gbits/sec    0   1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.69 GBytes  1.45 Gbits/sec    0             sender
[  5]   0.00-10.02  sec  1.69 GBytes  1.45 Gbits/sec                  receiver

iperf Done.
```





```
root@lab17:~ # uname -a
FreeBSD lab17 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@lab17:~ #

root@lab18:~ # uname -a
FreeBSD lab18 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@lab18:~ #
```


Bridged Interfaces: server to server

```
Accepted connection from 10.0.11.18, port 61273
[  5] local 10.0.11.17 port 5201 connected to 10.0.11.18 port 17033
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   310 MBytes  2.60 Gbits/sec
[  5]   1.00-2.00   sec   551 MBytes  4.62 Gbits/sec
[  5]   2.00-3.00   sec   589 MBytes  4.94 Gbits/sec
[  5]   3.00-4.00   sec   591 MBytes  4.96 Gbits/sec
[  5]   4.00-5.00   sec   592 MBytes  4.97 Gbits/sec
[  5]   5.00-6.00   sec   594 MBytes  4.99 Gbits/sec
[  5]   6.00-7.00   sec   594 MBytes  4.98 Gbits/sec
[  5]   7.00-8.00   sec   593 MBytes  4.98 Gbits/sec
[  5]   8.00-9.00   sec   593 MBytes  4.97 Gbits/sec
[  5]   9.00-10.00  sec   593 MBytes  4.98 Gbits/sec
[  5]  10.00-10.00  sec   337 KBytes  4.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.11.17
Connecting to host 10.0.11.17, port 5201
[  5] local 10.0.11.18 port 17033 connected to 10.0.11.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   312 MBytes  2.61 Gbits/sec   28    390 KBytes
[  5]   1.00-2.00   sec   551 MBytes  4.63 Gbits/sec   78    254 KBytes
[  5]   2.00-3.00   sec   589 MBytes  4.94 Gbits/sec   27    191 KBytes
[  5]   3.00-4.00   sec   591 MBytes  4.96 Gbits/sec   32    335 KBytes
[  5]   4.00-5.00   sec   592 MBytes  4.96 Gbits/sec   28    304 KBytes
[  5]   5.00-6.00   sec   595 MBytes  4.99 Gbits/sec   29    275 KBytes
[  5]   6.00-7.00   sec   593 MBytes  4.98 Gbits/sec   28    238 KBytes
[  5]   7.00-8.00   sec   593 MBytes  4.98 Gbits/sec   33    366 KBytes
[  5]   8.00-9.00   sec   593 MBytes  4.97 Gbits/sec   26    348 KBytes
[  5]   9.00-10.00  sec   593 MBytes  4.98 Gbits/sec   31    315 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec  340             sender
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec                  receiver

iperf Done.
```


Direct Interfaces: server to server

```
Accepted connection from 10.0.12.18, port 24603
[  5] local 10.0.12.17 port 5201 connected to 10.0.12.18 port 30690
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   997 MBytes  8.36 Gbits/sec
[  5]   1.00-2.00   sec  1.82 GBytes  15.6 Gbits/sec
[  5]   2.00-3.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   3.00-4.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   4.00-5.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   5.00-6.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   6.00-7.00   sec  1.83 GBytes  15.8 Gbits/sec
[  5]   7.00-8.00   sec  1.84 GBytes  15.8 Gbits/sec
[  5]   8.00-9.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   9.00-10.00  sec  1.83 GBytes  15.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.12.17
Connecting to host 10.0.12.17, port 5201
[  5] local 10.0.12.18 port 30690 connected to 10.0.12.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   998 MBytes  8.37 Gbits/sec   16    386 KBytes
[  5]   1.00-2.00   sec  1.82 GBytes  15.6 Gbits/sec   38    375 KBytes
[  5]   2.00-3.00   sec  1.83 GBytes  15.7 Gbits/sec   16    379 KBytes
[  5]   3.00-4.00   sec  1.83 GBytes  15.7 Gbits/sec   11   1.41 KBytes
[  5]   4.00-5.00   sec  1.83 GBytes  15.7 Gbits/sec   16    205 KBytes
[  5]   5.00-6.00   sec  1.83 GBytes  15.7 Gbits/sec    8    212 KBytes
[  5]   6.00-7.00   sec  1.83 GBytes  15.8 Gbits/sec    9    239 KBytes
[  5]   7.00-8.00   sec  1.84 GBytes  15.8 Gbits/sec    8    267 KBytes
[  5]   8.00-9.00   sec  1.83 GBytes  15.7 Gbits/sec    7    289 KBytes
[  5]   9.00-10.00  sec  1.83 GBytes  15.7 Gbits/sec    6    314 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec  135             sender
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec                  receiver

iperf Done.
```


jail to jail on same server

```
Accepted connection from 10.0.11.117, port 31176
[  5] local 10.0.11.127 port 5201 connected to 10.0.11.117 port 38861
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   203 MBytes  1.71 Gbits/sec
[  5]   1.00-2.00   sec   342 MBytes  2.87 Gbits/sec
[  5]   2.00-3.00   sec   343 MBytes  2.88 Gbits/sec
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   4.00-5.00   sec   224 MBytes  1.88 Gbits/sec
[  5]   5.00-6.00   sec   233 MBytes  1.96 Gbits/sec
[  5]   6.00-7.00   sec   241 MBytes  2.02 Gbits/sec
[  5]   7.00-8.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   8.00-9.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   9.00-10.00  sec   243 MBytes  2.04 Gbits/sec
[  5]  10.00-10.00  sec   851 KBytes  3.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec                  receiver
-----------------------------------------------------------

test-117 ~ # iperf3 -c 10.0.11.127
Connecting to host 10.0.11.127, port 5201
[  5] local 10.0.11.117 port 38861 connected to 10.0.11.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   204 MBytes  1.71 Gbits/sec    0    495 KBytes
[  5]   1.00-2.00   sec   342 MBytes  2.87 Gbits/sec    0    995 KBytes
[  5]   2.00-3.00   sec   343 MBytes  2.88 Gbits/sec    0   1.38 MBytes
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec    0   1.67 MBytes
[  5]   4.00-5.00   sec   224 MBytes  1.88 Gbits/sec    0   1.77 MBytes
[  5]   5.00-6.00   sec   233 MBytes  1.96 Gbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   241 MBytes  2.02 Gbits/sec    0   1.77 MBytes
[  5]   7.00-8.00   sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
[  5]   8.00-9.00   sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
[  5]   9.00-10.00  sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec                  receiver

iperf Done.
```


jail to jail on different servers

```
Accepted connection from 10.0.12.128, port 25861
[  5] local 10.0.12.127 port 5201 connected to 10.0.12.128 port 18803
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   221 MBytes  1.85 Gbits/sec
[  5]   1.00-2.00   sec   340 MBytes  2.85 Gbits/sec
[  5]   2.00-3.00   sec   344 MBytes  2.89 Gbits/sec
[  5]   3.00-4.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec
[  5]   5.00-6.00   sec   262 MBytes  2.20 Gbits/sec
[  5]   6.00-7.00   sec   390 MBytes  3.27 Gbits/sec
[  5]   7.00-8.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   8.00-9.00   sec   397 MBytes  3.33 Gbits/sec
[  5]   9.00-10.00  sec   398 MBytes  3.34 Gbits/sec
[  5]  10.00-10.00  sec  1014 KBytes  4.11 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec                  receiver
-----------------------------------------------------------

test-128 ~ # iperf3 -c 10.0.12.127
Connecting to host 10.0.12.127, port 5201
[  5] local 10.0.12.128 port 18803 connected to 10.0.12.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   221 MBytes  1.86 Gbits/sec    0    495 KBytes
[  5]   1.00-2.00   sec   340 MBytes  2.85 Gbits/sec    0   1007 KBytes
[  5]   2.00-3.00   sec   344 MBytes  2.89 Gbits/sec    0   1.47 MBytes
[  5]   3.00-4.00   sec   392 MBytes  3.29 Gbits/sec    0   1.77 MBytes
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec    0   1.77 MBytes
[  5]   5.00-6.00   sec   262 MBytes  2.20 Gbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   390 MBytes  3.27 Gbits/sec    0   1.77 MBytes
[  5]   7.00-8.00   sec   392 MBytes  3.29 Gbits/sec    0   1.77 MBytes
[  5]   8.00-9.00   sec   397 MBytes  3.33 Gbits/sec    0   1.77 MBytes
[  5]   9.00-10.00  sec   399 MBytes  3.34 Gbits/sec    0   1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec                  receiver

iperf Done.
```


----------

