# Thank you Armin Moradi for "FreeBSD jails and vnet from scratch."



## robroy (Jun 28, 2022)

FreeBSD Fans,

This weekend I was able to move some networking stuff from a bhyve VM in to a VNET jail, and I've been enjoying a fantastic performance improvement ever since.  VNET seems quite magical to me.

Part of what preventing prevented me from doing this earlier was that I also used sysutils/vm-bhyve on the same computer, and I couldn't figure out how to gracefully attach both bhyve VMs and the VNET jail to the same bridge instance.  So, "Combining bhyve and vnet networks" (in Mr. Moradi's article) got me pointed in the right direction.

I combed through his Web page but couldn't find any contact information.  So I figured I'd put my thank-you here on the forums.

Thank you Armin Moradi.

FreeBSD jails and vnet from scratch


----------



## Phishfry (Jun 29, 2022)

Can I ask why you don't use separate Ethernet adapters for your VNET Jails? Passthru.
Instead of bridge and epairs just slap a 2/4 port ethernet card in and tie it into your network.
You have switches correct? Same with bhyve. Passthru interfaces are my method.
It makes for a wiring mess but you get full bandwidth compared to a bridge.

I do like building from scratch jail.conf. It really isn't hard. I don't remember using the jib script though.


----------



## Phishfry (Jun 29, 2022)

I mean I look at this and laugh. Mine is like 4 lines.
The amount of contortions required for bridging is astounding.

```
# setup vnet
vnet;
vnet.interface = "epair${ep}b";

exec.prestart  = "ifconfig epair${ep} create";
exec.prestart += "ifconfig $bridge addm epair${ep}a";
exec.prestart += "ifconfig epair${ep}a ether $(/mpool/scripts/derive_mac.sh $iface $name 0 a)";
exec.prestart += "ifconfig epair${ep}b ether $(/mpool/scripts/derive_mac.sh $iface $name 0 b)";

exec.poststop  = "ifconfig $bridge deletem epair${ep}a";
exec.poststop += "ifconfig epair${ep}a destroy";
```


----------



## zirias@ (Jun 29, 2022)

Phishfry said:


> Instead of bridge and epairs just slap a 2/4 port ethernet card in and tie it into your network.
> You have switches correct? Same with bhyve. Passthru interfaces are my method.
> It makes for a wiring mess but you get full bandwidth compared to a bridge.


I prefer to bundle all available physical interfaces with lagg(4) (using LACP, of course the switch must support that). For things that must stay separated, vlan(4)s on top. For virtual machines and VNET jails, use bridge(4)s on these vlans.

Result: Full bandwidth, but used with full flexibility.


----------



## Phishfry (Jun 29, 2022)

That is my next attempt. lagg0 to jails.


			https://clinta.github.io/freebsd-jails-the-hard-way/


----------



## Phishfry (Jun 29, 2022)

The real unspoken hero of jails is disk speed.
Compared to bhyves 50%+ hit it really is refreshing to see your NVMe pulling same speed as on host.


----------



## Phishfry (Jun 29, 2022)

Might I add that adding jails via bsdinstall is sublime.
`bsdinstall jail /here/is/the/jail`


----------



## robroy (Jun 29, 2022)

Thanks Phishfry and Zirias.



Phishfry said:


> Can I ask why you don't use separate Ethernet adapters for your VNET Jails? Passthru.
> Instead of bridge and epairs just slap a 2/4 port ethernet card in and tie it into your network.



Yeah; using a dedicated network port for VNET jails is also recommended in FreeBSD Mastery: Jails by Michael W. Lucas, so this did cross my mind 'cause of that.  But I didn't know about the ease of configuration aspect.

I'm using a ThinkCentre M920q Tiny with its one onboard I219-LM port right now (also ruling out SR-IOV, I guess).

I remember seeing that there's an option to add an I350-T4 using a Lenovo-specific bracket.  That is something I could try to pick up, although I don't really need more total bandwidth than one gigabit connections' worth.



Phishfry said:


> You have switches correct?



Right now I only have an unmanaged switch at my desk, so LACP wouldn't be an option 'til I bought a new switch.  The switch is also uplinked to my main switch using a single gigabit link, so that'd be a bottleneck even if I added LACP at my desk.



Phishfry said:


> Same with bhyve. Passthru interfaces are my method.
> It makes for a wiring mess but you get full bandwidth compared to a bridge.



'makes sense.  The bandwidth part's not so much of a concern here since most of the traffic for this computer's bottlenecked by my Internet service.



Phishfry said:


> I do like building from scratch jail.conf. It really isn't hard. I don't remember using the jib script though.



I feel the same way.  I never felt motivated to learn about the jail tools in ports, because the tools in base (like jail(8)) seemed high-level enough to me; they seem more high-level than the stuff in base for bhyve (for bhyve, I definitely use sysutils/vm-bhyve).

I wouldn't have ever known about the jib script if not for FreeBSD Mastery: Jails by Michael W. Lucas.

I do like jib since it let me get a bridged VNET jail going much more quickly than I could have without it.  But it seems to have a drawback; it doesn't work with jails that have a hyphen in their name.  And I'd really like to use a hyphen in this particular jail's name.



Phishfry said:


> I mean I look at this and laugh. Mine is like 4 lines.
> The amount of contortions required for bridging is astounding.



Your config' does look great.  And without jib I totally agree about the bridging configuration.  I looked at it a few times over the years and thought, "uhh...  no thanks."  Plus I'd already invested a lot in to understanding the old-school, pre-VNET jail networking scheme and it worked for all of my uses, until recently.



Zirias said:


> I prefer to bundle all available physical interfaces with lagg(4) (using LACP, of course the switch must support that). For things that must stay separated, vlan(4)s on top. For virtual machines and VNET jails, use bridge(4)s on these vlans.
> 
> Result: Full bandwidth, but used with full flexibility.



Thanks; I like this configuration a lot and used it elsewhere once.  There's something fascinating about this LACP + VLANs set up.  I guess what surprises me the most about it is that it actually _works_.

I find that this computer's traffic tends to be bursty though.  Rarely does it have multiple sustained, simultaneous flows that'd distribute well with LACP.  And though LACP is undeniably _fun_ the truth is that I don't really need more bandwidth in this case.



Phishfry said:


> The real unspoken hero of jails is disk speed.
> Compared to bhyves 50%+ hit it really is refreshing to see your NVMe pulling same speed as on host.



Yes; I'm sure you're right about this!  I had been running X clients in the bhyve VM and displaying them on my local X server.  I found that the same X clients "felt" much snappier when running inside of the VNET jail, as opposed to bhyve.

I guess that's what happens with several complex layers are yanked out.  Not to mention the sudden access to all CPU threads and memory.  I love jails.

Here's the configuration I'm using now:

/etc/rc.conf

```
# See https://www.amoradi.org/20210908201936.html
cloned_interfaces="bridge0"
ifconfig_bridge0_name="em0bridge"
create_args_bridge0="addm em0"

jail_enable="YES"
```

/etc/jail.conf

```
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = /j/$name;

rgregg0a {
    vnet;
    vnet.interface="e0b_rgregg0a";
    exec.prestart+="/usr/local/bin/jib addm rgregg0a em0";
    exec.poststop+="/usr/local/bin/jib destroy rgregg0a";
}
```


----------



## robroy (Jun 29, 2022)

Related to jib:  I actually had the chance to ask its author (Devin Teske) about VNET stuff in person at MeetBSD 2016 in Berkeley, California.

'good times.


----------

