# Multiple Nics, Networks, VNET and Iocage



## zader (Oct 23, 2021)

I recently did some fairly complex networking for a project and thought .. hey maybe someone else may find it useful....

I assume the following:

multiple network cards
freebsd 13
iocage with Vnets
---
This KB scales with how ever many network cards you have..  it assumes (2) igb0 and igb1 .. if you have 6 then just add more .. likewise if you only have 1.. remove anything wiht igb1 in it.

You should end up with 4 jails.
*igb0-wan* with vnet0 using DHCP and vnet1 configured as the 10.10.10.1 gateway and 1 jail *igb0-application* at 10.10.10.100
*igb1-wan* with vnet0 using DHCP and vnet1 configured as the 172.16.24.1 gateway and 1 jail *igb1-application *at 172.16.24.100


/etc/sysctl.conf

```
net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0
```
adds some back end plumbing

/etc/rc.conf

```
iocage_enable="YES"

# enable pf and configure a path for your rule set
pf_enable="YES"
pflog_enable="YES"
pf_rules="/etc/pf.conf"

# allow the host to act as a gateway
gateway_enable="YES"

# rename your network cards
ifconfig_igb0_name="igb0wan"
ifconfig_igb1_name="igb1wan"
# bring them up
ifconfig_igb0wan="up"
ifconfig_igb1wan="up"

# create your internal and external bridges
cloned_interfaces="bridge0 bridge1 bridge2 bridge3"

# name your bridges so you can attach them properly ie public vs private
ifconfig_bridge0_name="igb0wanbridge"
ifconfig_bridge1_name="igb1wanbridge"
ifconfig_bridge2_name="igb0pbridge"
ifconfig_bridge3_name="igb1pbridge"

# bring your bridges up
ifconfig_igb0wanbridge="addm igb0wan up"
ifconfig_igb1wanbridge="addm igb1wan up"
```

*NOTE:* if you had 3 network cards, just copy these 4 lines and change the interface id.. ir igb4

```
ifconfig_igb0_name="igb0wan"
ifconfig_igb0wan="up"
cloned_interfaces="bridge0 ...
ifconfig_bridge0_name="igb0wanbridge"
ifconfig_bridge2_name="igb0pbridge"
```

normally just a service restart will do, *but I would reboot at this point* to make sure.

upon restart you should see something like this ...

`ifconfig`
interfaces should be renamed to *igb0wan/igb1wan*

```
igb0wan: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether xxxxxxxxxxxx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1wan: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
ether xxxxxxxxxxxxx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
... cut
4 new bridges, *igb0wanbridge, igb0pbridge, igb1wanbridge, igb1pbridge*
aka 2 public sides and 2 private sides

```
igb0wanbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0wan flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1wanbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb1wan flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb0pbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether xxxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1pbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether xxxxxxxxxxxxx
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
```

igb0wanbridge -- the front side of the bridge
igb0pbridge -- the private side
at this point there are no jails associated to any of the bridges

*Let’s create the WAN/LAN gateway*

the gateway jails *igb0-wan/igb1-wan *basically have 2 network cards in them, the first network adaptor connects to the WAN and the other to the LAN.  Just like any normal firewall device. This allows you to route, block, forward or scrub traffic just like any other firewall.  It also allows you to tag your internal application traffic.

*.172 Network WAN*

```
iocage create -r 13.0-RELEASE priority=50 boot=on allow_raw_sockets=1 vnet=on bpf=1 defaultrouter=10.100.10.1 ip4_addr="vnet0|dhcp,vnet1|172.16.24.1/24" interfaces="vnet0:igb0wanbridge,vnet1:igb0pbridge" resolver=10.100.10.1 -n igb0-wan
```

*priority=50* raises the boot up priority to ensure this jail comes up first
*boot=on* tells the host to bring the jail up on boot up
*allow_raw_sockets=1* enables the gateway jail to use pf
*bpf* = required to use dhcp
*defaultrouter= *(generally your network gateway)
*ip4_addr* = vnet0 uses dhcp to fetch an ip, vnet1 assigns the gateway ip you want to use for your internal network
*interfaces* = vnet0: bind to bridge device (in this case its the wan/internet side of the bridge) vnet1 is the other end that attaches to the internal network
*resolver* = what ever ip you want to use for dns
*-n* name of jail

In my case I am using to physical cables, one plugged into igb0 that goes to network 1 and the other into igb1 that goes to network 2... they are totally separate so that I can destory and rebuild them at will without touching anything else.

In theory if you had 10 nics, you could aslo build lags more bridges and tunnels galore.

*.10 Network WAN*

```
iocage create -r 13.0-RELEASE priority=51 boot=on allow_raw_sockets=1 vnet=on bpf=1 defaultrouter=10.100.10.1 ip4_addr="vnet0|dhcp,vnet1|10.10.10.1/24" interfaces="vnet0:igb1wanbridge,vnet1:igb1pbridge" resolver=10.100.10.1 -n igb1-wan
```


this is the command for the wan jail on igb1.. I’ve included it for comparison , if you want more .. up arrow and change the good bits (assuming you created more bridges in the /etc/rc.conf above

*Now lets add a jail to the private network

.172 Network Application*

```
iocage create -r 13.0-RELEASE priority=60 boot=on vnet=on defaultrouter=172.16.24.1 ip4_addr="vnet0|172.16.24.100/24" interfaces="vnet0:igb0pbridge" resolver=172.16.24.1 -n igb0-application
```

*.10 Network Application*

```
iocage create -r 13.0-RELEASE  priority=61 boot=on vnet=on defaultrouter=10.10.10.1 ip4_addr="vnet0|10.10.10.100/24" interfaces="vnet0:igb1pbridge" resolver=10.10.10.1 -n igb1-application
```

there are some minor differences, such as the boot priority, the routers use the wan-jails ip address and the local lan ip is 124.100 likewise the resolver is set to the upstream wan's internal ip.

Lets bring it up!

`iocage list`

```
+-----+------------------+-------+--------------+------------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+==================+=======+==============+==================+
| 7 | igb0-application | up | 13.0-RELEASE | 172.16.24.100 |
+-----+------------------+-------+--------------+------------------+
| 5 | igb0-wan | up | 13.0-RELEASE | dhcp,172.16.24.1 |
+-----+------------------+-------+--------------+------------------+
| 8 | igb1-application | up | 13.0-RELEASE | 10.10.10.100 |
+-----+------------------+-------+--------------+------------------+
| 6 | igb1-wan | up | 13.0-RELEASE | dhcp,10.10.10.1 |
+-----+------------------+-------+--------------+------------------+
```
when you do an ifconfig on the host you should see all of your vnet devices..

`ifconfig`
.... cut sample.

```
vnet0.8: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: igb1-application as nic: epair0b
options=8<VLAN_MTU>
ether xxxxxxxxxxx
hwaddr 02:23:95:84:97:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

presto..

*Testing*

`iocage console igb0-wan`  (aka the .172 network)
the WAN host can ping the internet,its gateway, or hosts on the 172 network


```
igb0-wan:~ # ping 10.100.10.1
PING 10.100.10.1 (10.100.10.1): 56 data bytes
64 bytes from 10.100.10.1: icmp_seq=0 ttl=255 time=0.256 ms
root@igb0-wan:~ # ping cnn.com
PING 151.101.65.67 (151.101.65.67): 56 data bytes
64 bytes from 151.101.65.67: icmp_seq=0 ttl=255 time=0.230 ms
root@igb0-wan:~ # ping 172.16.24.1
PING 172.16.24.1 (172.16.24.1): 56 data bytes
64 bytes from 172.16.24.1: icmp_seq=0 ttl=64 time=0.145 ms
root@igb0-wan:~ # ping 172.16.24.100
PING 172.16.24.100 (172.16.24.100): 56 data bytes
64 bytes from 172.16.24.100: icmp_seq=0 ttl=64 time=0.252 ms
```


*Additional Notes:*

In my case this entire project was to simply make something that can be built, managed and destroyed by ansible without touching anything else.  I thought it would be a good learing tool to share and may help others.

please feel free to reply with your thoughts & comments, updates, or additions.

Cheers!


----------

