# Max packet size less then MTU value on direct link with BCM5715C



## ro (Sep 23, 2011)

Hi all.

Sorry for my english, it is not native for me.

I've three devices:
- C2801 (c2801-adventerprisek9-mz.124-3h);
- C2960 (c2960-lanbase-mz.122-35.SE5);
- Sun Fire X2100 M2 (FreeBSD 8.1-RELEASE amd64);

They are connected so:
c2801-fa0/1 <--> fa0/24-c2960-fa0/3 <--> bge0-Sun_Fire_X2100_M2 in vlan id 17 802.1q
bge0 is Broadcom BCM5715C 10/100/100 PCIe Ethernet Controller.
cisco 2801 is default router for FreeBSD in vlan17 & has ip XXX.YYY.254.33

MTU on FreeBSD interfaces:

```
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:1b:24:5d:38:1f
        media: Ethernet 100baseTX <full-duplex>
        status: active
[root@sf1 /usr/home/sudosu]# ifconfig vlan17
vlan17: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:1b:24:5d:38:1f
        inet XXX.YYY.254.35 netmask 0xfffffff0 broadcast XXX.YYY.254.47
        media: Ethernet 100baseTX <full-duplex>
        status: active
        vlan: 17 parent interface: bge0
[root@sf1 /usr/home/sudosu]#
```
System MTU and Fa0/3, Fa0/24 MTU on catalyst:

```
C2960-24-SPB-SRV1#sh sys MTU

System MTU size is 1500 bytes
System Jumbo MTU size is 1500 bytes
Routing MTU size is 1500 bytes
C2960-24-SPB-SRV1#sh int fa0/3 mtu

Port      Name               MTU
Fa0/3     "sf1.tech bge0 tru 1500        
C2960-24-SPB-SRV1#sh int fa0/24 mtu

Port      Name               MTU
Fa0/24    "Uplink 2800-SPB-S 1500        
C2960-24-SPB-SRV1#
```
MTU on fa0/1.17 on c2801

```
C2800-SPB-SRV#sh int fa0/1 | i MTU
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, 
C2800-SPB-SRV#sh int fa0/1.17 | i MTU
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, 
C2800-SPB-SRV#
```
Try to ping cisco 2801 from freebsd one whith df bit. IMHO maximum packed size should be 1500 - 20 (ip and tcp header) - 8 (icmp header) = 1472.
But I see:

```
[root@sf1 /usr/home/sudosu]# ping -D -s 1273 -c 1 XXX.YYY.254.33
PING XXX.YYY.254.33 (XXX.YYY.254.33): 1273 data bytes
ping: sendto: Message too long

--- XXX.YYY.254.33 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
[root@sf1 /usr/home/sudosu]# ping -D -s 1272 -c 1 XXX.YYY.254.33
PING XXX.YYY.254.33 (XXX.YYY.254.33): 1272 data bytes
1280 bytes from XXX.YYY.254.33: icmp_seq=0 ttl=255 time=1.181 ms

--- XXX.YYY.254.33 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.181/1.181/1.181/0.000 ms
[root@sf1 /usr/home/sudosu]#
```

Can anybody explain me - where is 200 bytes?
Has anybody some ideas how to fix this situation?


----------



## ro (Sep 23, 2011)

This MTU troubles make impossible calculating right mss value to tcp session. All tcp connections, with large packets and df-bit are breaks whith time out


----------



## yongari@ (Sep 26, 2011)

Hmm, I'm not aware of any VLAN oversize frame issues on bge(4) controllers.
Please check whether netstat(1) also reports counter for "datagrams that can't be fragmented". Also use tcpdump(1) on parent interface (bge) and check whether you can see ICMP needfrag packet which could be sent from remote router for your ICMP echo request.


----------



## ro (Sep 27, 2011)

10x yongari@

I've checked icmp packages on parent (bge0) interface.
While the size of packages is less 1273 they go in both directions. Else only from router to FreeBSD direction and not another one.


----------



## ro (Sep 27, 2011)

> ...
> I've checked icmp packages on parent (bge0) interface.
> ...


With tcpdump.
How can I see netstat counters?


----------



## yongari@ (Sep 28, 2011)

Send larger packet than 1273 with 'Don't fragment bit set' from FreeBSD box and see which one(FreeBSD box or router) blocks the packet. As I said, running tcpdump on FreeBSD box can show you which one dropped the packet. If you can't see any output packet sent from FreeBSD with the tcpdump, it indicates FreeBSD is dropping the packet.


----------



## yongari@ (Sep 28, 2011)

Just run netstat command.


----------

