# MTU of jail vimage by netgraph



## epopen (Aug 30, 2022)

Hi All

I have a question about MTU of jail vimage by netgraph.
Please reference illustration of running system as follows.




Running routing mode with *gateway_enable="YES"* @ /etc/rc.conf.

Connect *Jail of ssh* from *netif msk0*(internet) via ssh.
Log in is OK, but disconnect unexpected when high data traffic (example: list directory/file @ `ls`).

All work fine when all of MTU=1492

Result of`ifconfig ng0`

```
ng_vimage0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
```
And `ifconfig ng0_sshd`

```
ng0_sshd: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
```
Both have options _JUMBO_MTU, _it added automatically either MTU 1492 or 9000.
Jumbo frame support I suppose.

My question:

Is netgraph does not support jumbo frame?
If above true, problem caused by packet fragmentation/de-fragmentation,
Is configure require or other?
Because host <-> jail @ IPv6 only and _IPv6 *routers* do not fragment IPv6 packets_,
Can not use jumbo frame with IPv6 meant?
Missing some else?

Addition information about MTU of bridge
In https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf
_"MTU increased to 9000, allowing large numbers of neighbors..."_ written
And example code also as follow
`ifconfig bridge create name vnetdemobridge mtu 9000 up`
The case is if_bridge(), I does not find this kind of example for ng_bridge() with my case.


Sorry many question.
Thanks a lot.


----------



## VladiBG (Aug 30, 2022)

Does your firewall allowing ICMP?



> Many network security devices block all ICMP messages for perceived security benefits, including the errors that are necessary for the proper operation of PMTUD. This can result in connections that complete the TCP three-way handshake correctly, but then hang when data are transferred. This state is referred to as a _black hole connection_.[7]








						Path MTU Discovery - Wikipedia
					






					en.wikipedia.org


----------



## skeletor (Aug 30, 2022)

epopen  ,
 you can't use Jumbo frames with interfaces, which hasn't suitable for this. It's wrong, and can cause some network troubles. You can't create bridge with Jumbo from interfaces, which isn't configured with Jumbo.
Jumbo uses only within LAN or within channels with only clear ethernet proto. It's not suitable for use with pptp, l2tp,
Also, you can remembered, that Jumbo MUST be configured at all devices, which operate this traffic: switches, NICs, routers,...

Is netgraph does not support jumbo frame? === Netgraph uses to connect interfaces, may be to forward packets, not to operate Jumbo
May be, you don't understand using Jumbo. It's uses when you have high speed network, and you want to increase speed. Jumbo assumes, that packets don't be fragmented during transfer from one host to other if they have size less Jumbo size. But, if some transition host doesn't accept Jumbo, your packets will be fragmented according to size this host MTU. So, if you try to setup Jumbo over MTU 1492, your 9000 packet will be chunked to packets with 1470 size (MTU minus overhead proto).


----------



## epopen (Aug 31, 2022)

VladiBG said:


> Does your firewall allowing ICMP?
> 
> 
> 
> ...


Thanks you a lot.

From jail, can be ping other jail each other and host(interface *ng_vimage0* and *ng0,*
But can not ping internet currently, can be ping internet after firewall adjustment.


----------



## epopen (Aug 31, 2022)

skeletor said:


> epopen  ,
> you can't use Jumbo frames with interfaces, which hasn't suitable for this. It's wrong, and can cause some network troubles. You can't create bridge with Jumbo from interfaces, which isn't configured with Jumbo.
> Jumbo uses only within LAN or within channels with only clear ethernet proto. It's not suitable for use with pptp, l2tp,
> Also, you can remembered, that Jumbo MUST be configured at all devices, which operate this traffic: switches, NICs, routers,...
> ...


Thanks you a lot, your explain clarity.

Because there is data exchange between jail.
My consider was that using jumbo frame between jail would reduce the overhead fragmentation/de-fragmentation.
Therefore jumbo frame are only used between jail.

I assume.

Virtual interface with physical interface similar.
Packet fragmentation at host if packets from MTU=9000 of jail interface *ng_vimage0* to MTU=1492 of internet interface *ng0*.
All of jail interface *ng_vimage0/ng0_sshd *have *options of JUMBO_MTU,* therefore it can be jumbo frame.


----------



## skeletor (Aug 31, 2022)

I'm a newbie in jail, but if it possible to use a communication via loopback interface between jails it would be the fastest way. Why? Loopback support a Jumbo, loopback doesn't use a routing. 

If impossible use loopback, if it applicable for you, you can switch off the checksum check by ifconfig -RXCSUM em0 (or TXCSUM, note, if you disable checksum you also need to disable TSO4, TSO6 on interface). Additionaly, you can tuning your NICs via sysctl: queue size (below example for Intel "em" cards)


```
/boot/loader.conf:
hw.em.rxd=4096
hw.em.txd=4096
hw.em.max_interrupt_rate=32000

/etc/sysctl.conf:
dev.em.0.rx_int_delay=200
dev.em.0.tx_int_delay=200
dev.em.0.rx_abs_int_delay=4000
dev.em.0.tx_abs_int_delay=4000
dev.em.0.rx_processing_limit=4096
```


----------



## epopen (Sep 1, 2022)

skeletor said:


> I'm a newbie in jail, but if it possible to use a communication via loopback interface between jails it would be the fastest way. Why? Loopback support a Jumbo, loopback doesn't use a routing.
> 
> If impossible use loopback, if it applicable for you, you can switch off the checksum check by ifconfig -RXCSUM em0 (or TXCSUM, note, if you disable checksum you also need to disable TSO4, TSO6 on interface). Additionaly, you can tuning your NICs via sysctl: queue size (below example for Intel "em" cards)
> 
> ...


Thanks a lot.

In fact, because network isolation, I migrated from conventional(cloned lo1) to vimage before.

About checksum, I executed as follows.
`ifconfig ng_vimage0 rxcsum`
and
`ifconfig ng_vimage0 -rxcsum`
Both no error, but not affect(options never change) as follows.

```
ng_vimage0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 02:ac:95:e3:83:d2
        hwaddr 58:9c:fc:00:17:0a
        inet6 fd00::ffff:a00:3fe prefixlen 119
        inet6 fe80::ac:95ff:fee3:83d2%ng_vimage0 prefixlen 64 scopeid 0x6
        inet 10.0.3.254 netmask 0xfffffe00 broadcast 10.0.3.255
        fib: 1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=61<PERFORMNUD,AUTO_LINKLOCAL,NO_RADR>
```

It affect physical interface only look like.


----------

