# Packet delays



## nicolas (Nov 3, 2009)

Hello!
We have a problem...
Dlink 3426 is core of our network. DGS 3612 - routers, which are connected to core (users). Core is connected to bridge. There are three interfaces igb0, igb1, igb2 on bridge:
igb0 - local network
igb1 - border1
igb2 - border2
Bridge is are shaper (ipfw). border1 and border2 are connected to bridge.
Border1:

```
FreeBSD 7.0-STABLE-200807 i386
```
Border2:

```
FreeBSD 7.2-STABLE-200906 i386
```
Bridge:

```
FreeBSD 7.2-STABLE-200906 amd64
```
Traffic is devided between two borders. We have not got any problems with border2. But on border1 we have: ping through border1 from users and from bridge

```
ping ya.ru
PING ya.ru (93.158.134.8): 56 data bytes
64 bytes from 93.158.134.8: icmp_seq=0 ttl=58 time=95.083 ms
64 bytes from 93.158.134.8: icmp_seq=1 ttl=58 time=102.876 ms
64 bytes from 93.158.134.8: icmp_seq=2 ttl=58 time=113.630 ms
64 bytes from 93.158.134.8: icmp_seq=3 ttl=58 time=90.352 ms
64 bytes from 93.158.134.8: icmp_seq=4 ttl=58 time=127.349 ms
64 bytes from 93.158.134.8: icmp_seq=5 ttl=58 time=115.942 ms
64 bytes from 93.158.134.8: icmp_seq=6 ttl=58 time=103.753 ms
64 bytes from 93.158.134.8: icmp_seq=7 ttl=58 time=124.507 ms
```
But in  border1 we have:

```
ping ya.ru
PING ya.ru (213.180.204.8): 56 data bytes
64 bytes from 213.180.204.8: icmp_seq=0 ttl=61 time=3.130 ms
64 bytes from 213.180.204.8: icmp_seq=1 ttl=61 time=2.909 ms
64 bytes from 213.180.204.8: icmp_seq=2 ttl=61 time=3.791 ms
64 bytes from 213.180.204.8: icmp_seq=3 ttl=61 time=2.953 ms
64 bytes from 213.180.204.8: icmp_seq=4 ttl=61 time=3.298 ms
64 bytes from 213.180.204.8: icmp_seq=5 ttl=61 time=3.096 ms
```
bridge:
1).

```
netstat -w1d -I igb0
			input		 (igb0)		   output
   packets  errs	  bytes	packets  errs	  bytes colls
	 32442	 0   19375201	  29681	 0   23220291	 0
	 31894	 0   19334182	  29124	 0   23043942	 0
	 31566	 0   18885017	  28558	 0   22390016	 0
	 31810	 0   19200993	  28768	 0   22270687	 0
	 31879	 0   19347245	  29145	 0   22795440	 0
	 31697	 0   18997706	  29274	 0   22651927	 0
	 32042	 0   18963695	  29634	 0   23508103	 0
	 30674	 0   18123997	  28432	 0   22890282	 0
	 31654	 0   18519433	  28860	 0   22918799	 0
	 31961	 0   19145696	  29418	 0   23238031	 0
	 32056	 0   19053994	  29723	 0   23775215	 0
	 32367	 0   18952231	  29771	 0   23774054	 0
```
2).

```
netstat -w1d -I igb1
			input		 (igb1)		   output
   packets  errs	  bytes	packets  errs	  bytes colls
	 21378	 0   19903174	  21410	 0	9866489	 0
	 21444	 0   19948761	  21752	 0   10266869	 0
	 21435	 0   19972984	  21825	 0   10289265	 0
	 21318	 0   19748715	  21431	 0   10238696	 0
	 21606	 0   19855988	  21467	 0   10483023	 0
	 21825	 0   19956737	  21750	 0   10452641	 0
	 21665	 0   19805016	  21980	 0   10711357	 0
	 20937	 0   18983521	  21357	 0   10506764	 0
	 21241	 0   19280484	  21992	 0   10713008	 0
	 21469	 0   19652542	  21667	 0   10391424	 0
	 21235	 0   19538130	  21524	 0   10757572	 0
	 20779	 0   18963762	  21003	 0   10525245	 0
	 20853	 0   18854491	  21312	 0   10559890	 0
```
Those commands on border1:
1). internal

```
netstat -w1d -I em1
			input		  (em1)		   output
   packets  errs	  bytes	packets  errs	  bytes colls
	  9862	 0	8286463	   8226	 0	4172860	 0
	 10060	 0	8603460	   7209	 0	3184874	 0
	  9290	 0	7781648	   7709	 0	3672720	 0
	  9720	 0	8311391	   7208	 0	3545028	 0
	 10256	 0	8669129	   8232	 0	4678531	 0
	  9509	 0	8140150	   7715	 0	3878637	 0
	  9554	 0	8081766	   7199	 0	3512477	 0
	  9426	 0	8046217	   7721	 0	3867273	 0
	  9795	 0	8177803	   7721	 0	3729722	 0
	  9637	 0	8297993	   7199	 0	3766607	 0
	  9671	 0	8275210	   7202	 0	3510855	 0
	  9451	 0	8306122	   7196	 0	3535604	 0
	  9324	 0	8009044	   7202	 0	3092661	 0
	  9715	 0	8265237	   7364	 0	3575433	 0
	  9908	 0	8215978	   7499	 0	3911766	 0
	  9776	 0	8270432	   8086	 0	4178572	 0
```
2). external

```
netstat -w1d -I em0
			input		  (em0)		   output
   packets  errs	  bytes	packets  errs	  bytes colls
	  8253	 0	4640677	   9803	 0	8495567	 0
	  7897	 0	4878726	  10041	 0	7322200	 0
	  8368	 0	4776220	   9832	 0	8565609	 0
	  7650	 0	4285985	   9283	 0	7717595	 0
	  7891	 0	4352473	   9797	 0	8134889	 0
	  8606	 0	5018218	   9657	 0	8289548	 0
	  8460	 0	4720658	  10207	 0	8397786	 0
	  7543	 0	4085187	   9277	 0	8059337	 0
	  9376	 0	6040976	  11500	 0	8860064	 0
	  8181	 0	4552212	  10295	 0	8582838	 0
	  8774	 0	5079266	  10062	 0	8494193	 0
	  8193	 0	4674008	  10295	 0	8520010	 0
	 10170	 0	7212068	  10813	 0	8596957	 0
	  8499	 0	4936674	  10828	 0	9090783	 0
	  8416	 0	4409607	  10297	 0	9050824	 0
	  7613	 0	3960242	   9280	 0	8231506	 0
```
A few month ago border1 in this configuration worked succsessfully..
On border1 there is yandex driver for em. 
loader.conf:

```
hw.em.rxd="4096"
hw.em.txd="4096"
```
/etc/sysctl.conf Ð½Ð° border1:

```
net.inet.tcp.recvspace=262144 
net.inet.tcp.sendspace=62144 
net.inet.udp.recvspace=1048576
kern.ipc.maxsockbuf=4194304 
kern.ipc.nmbclusters=262144 
net.inet.ip.portrange.first=5700
kern.ipc.somaxconn=65535 
net.inet.tcp.blackhole=2 
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=30
net.inet.ip.intr_queue_maxlen=4096 
net.inet.tcp.delayed_ack=0
net.inet.tcp.delacktime=10 
net.inet.tcp.newreno=0 
net.inet.tcp.msl=2500 
net.inet.ip.rtmaxcache=1024
net.inet.raw.recvspace=65536
net.inet.ip.dummynet.hash_size=65536
net.inet.ip.fw.dyn_ack_lifetime=60 
net.inet.ip.fw.dyn_syn_lifetime=10 
net.inet.ip.fw.dyn_fin_lifetime=10 
net.inet.ip.fw.dyn_max=16192
net.inet.ip.fastforwarding=1 
net.isr.direct=1 
net.inet.icmp.drop_redirect=1
dev.em.0.rx_kthreads=4
dev.em.1.rx_kthreads=4
dev.em.2.rx_kthreads=4
```
I have made the same options in sysctl on border1 like on border2:

```
kern.ipc.nmbclusters=262144
net.inet.icmp.icmplim=200
I have added a new option
net.inet.tcp.sendbuf_max=16777216 
net.inet.tcp.recvbuf_max=16777216 
net.bpf.bufsize=4194304
net.bpf.maxbufsize=8388608
```
But have no ane good results.

dmesg:
Border2 external:

```
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 0
em0: Receive No Buffers = 0
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 0
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 5018002561
em0: Good Packets Xmtd = 5133609771
em0: TSO Contexts Xmtd = 0
em0: TSO Contexts Failed = 0
```
Border1 internal:

```
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 9975
em0: Receive No Buffers = 7099
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 1
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 2639750787
em0: Good Packets Xmtd = 2455117624
em0: TSO Contexts Xmtd = 18
em0: TSO Contexts Failed = 0
```

Border2 internal:

```
em1: Excessive collisions = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 0
em1: Receive No Buffers = 0
em1: Receive Length Errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Collision/Carrier extension errors = 0
em1: RX overruns = 0
em1: watchdog timeouts = 0
em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 0
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 0
em1: Good Packets Rcvd = 5189211824
em1: Good Packets Xmtd = 4995412107
em1: TSO Contexts Xmtd = 16447
em1: TSO Contexts Failed = 0
```

Border1 internal:

```
em1: Excessive collisions = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 0
em1: Receive No Buffers = 0
em1: Receive Length Errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Collision/Carrier extension errors = 0
em1: RX overruns = 0
em1: watchdog timeouts = 0
em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 0
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 0
em1: Good Packets Rcvd = 2469872680
em1: Good Packets Xmtd = 2578357516
em1: TSO Contexts Xmtd = 14174
em1: TSO Contexts Failed = 0
```


----------



## nicolas (Nov 3, 2009)

Bridge:
internal

```
igb0: Adapter hardware address = 0xffffff00012f0528 
igb0: CTRL = 0xc00241 RCTL = 0x801a 
igb0: Packet buffer = Tx=0k Rx=0k 
igb0: Flow control watermarks high = 63488 low = 61988
igb0: Queue(0) tdh = 154, tdt = 154
igb0: no descriptors avail event = 0
igb0: TX(0) MSIX IRQ Handled = 26582786007
igb0: TX(0) Packets sent = 69466117227
igb0: Queue(1) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(1) MSIX IRQ Handled = 0
igb0: TX(1) Packets sent = 0
igb0: Queue(2) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(2) MSIX IRQ Handled = 0
igb0: TX(2) Packets sent = 0
igb0: Queue(3) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(3) MSIX IRQ Handled = 0
igb0: TX(3) Packets sent = 0
igb0: Queue(0) rdh = 186, rdt = 185
igb0: RX(0) Packets received = 18340736258
igb0: RX(0) Split Packets = 0
igb0: RX(0) Byte count = 12477314664354
igb0: RX(0) MSIX IRQ Handled = 11343806385
igb0: RX(0) LRO Queued= 0
igb0: RX(0) LRO Flushed= 0
igb0: Queue(1) rdh = 150, rdt = 149
igb0: RX(1) Packets received = 18360157016
igb0: RX(1) Split Packets = 0
igb0: RX(1) Byte count = 12474408387147
igb0: RX(1) MSIX IRQ Handled = 11362784395
igb0: RX(1) LRO Queued= 0
igb0: RX(1) LRO Flushed= 0
igb0: Queue(2) rdh = 142, rdt = 140
igb0: RX(2) Packets received = 18122235444
igb0: RX(2) Split Packets = 0
igb0: RX(2) Byte count = 12331139081653
igb0: RX(2) MSIX IRQ Handled = 11268633644
igb0: RX(2) LRO Queued= 0
igb0: RX(2) LRO Flushed= 0
igb0: Queue(3) rdh = 95, rdt = 94
igb0: RX(3) Packets received = 18379433903
igb0: RX(3) Split Packets = 0
igb0: RX(3) Byte count = 12407828772109
igb0: RX(3) MSIX IRQ Handled = 11387935869
igb0: RX(3) LRO Queued= 0
igb0: RX(3) LRO Flushed= 0
igb0: LINK MSIX IRQ Handled = 5
igb0: Mbuf defrag failed = 0
igb0: Std mbuf header failed = 0
igb0: Std mbuf packet failed = 0
igb0: Driver dropped packets = 0
igb0: Driver tx dma failure in xmit = 0
```

border1

```
igb1: Adapter hardware address = 0xffffff000451a528 
igb1: CTRL = 0x18c00241 RCTL = 0x801a 
igb1: Packet buffer = Tx=0k Rx=0k 
igb1: Flow control watermarks high = 63488 low = 61988
igb1: Queue(0) tdh = 150, tdt = 151
igb1: no descriptors avail event = 1
igb1: TX(0) MSIX IRQ Handled = 9053729033
igb1: TX(0) Packets sent = 15117561092
igb1: Queue(1) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(1) MSIX IRQ Handled = 0
igb1: TX(1) Packets sent = 0
igb1: Queue(2) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(2) MSIX IRQ Handled = 0
igb1: TX(2) Packets sent = 0
igb1: Queue(3) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(3) MSIX IRQ Handled = 0
igb1: TX(3) Packets sent = 0
igb1: Queue(0) rdh = 253, rdt = 251
igb1: RX(0) Packets received = 3887801598
igb1: RX(0) Split Packets = 0
igb1: RX(0) Byte count = 3419307814950
igb1: RX(0) MSIX IRQ Handled = 2880414195
igb1: RX(0) LRO Queued= 0
igb1: RX(0) LRO Flushed= 0
igb1: Queue(1) rdh = 119, rdt = 118
igb1: RX(1) Packets received = 3908297847
igb1: RX(1) Split Packets = 0
igb1: RX(1) Byte count = 3460826092933
igb1: RX(1) MSIX IRQ Handled = 2888114773
igb1: RX(1) LRO Queued= 0
igb1: RX(1) LRO Flushed= 0
igb1: Queue(2) rdh = 152, rdt = 151
igb1: RX(2) Packets received = 3922202264
igb1: RX(2) Split Packets = 0
igb1: RX(2) Byte count = 3503326896463
igb1: RX(2) MSIX IRQ Handled = 2886529770
igb1: RX(2) LRO Queued= 0
igb1: RX(2) LRO Flushed= 0
igb1: Queue(3) rdh = 92, rdt = 91
igb1: RX(3) Packets received = 3887004765
igb1: RX(3) Split Packets = 0
igb1: RX(3) Byte count = 3441233186193
igb1: RX(3) MSIX IRQ Handled = 2879712428
igb1: RX(3) LRO Queued= 0
igb1: RX(3) LRO Flushed= 0
igb1: LINK MSIX IRQ Handled = 10
igb1: Mbuf defrag failed = 0
igb1: Std mbuf header failed = 0
igb1: Std mbuf packet failed = 0
igb1: Driver dropped packets = 0
igb1: Driver tx dma failure in xmit = 0
```

Border2:

```
igb2: Adapter hardware address = 0xffffff0004575528 
igb2: CTRL = 0x18c00241 RCTL = 0x801a 
igb2: Packet buffer = Tx=0k Rx=0k 
igb2: Flow control watermarks high = 63488 low = 61988
igb2: Queue(0) tdh = 74, tdt = 74
igb2: no descriptors avail event = 3117
igb2: TX(0) MSIX IRQ Handled = 22305812305
igb2: TX(0) Packets sent = 57738555533
igb2: Queue(1) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(1) MSIX IRQ Handled = 0
igb2: TX(1) Packets sent = 0
igb2: Queue(2) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(2) MSIX IRQ Handled = 0
igb2: TX(2) Packets sent = 0
igb2: Queue(3) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(3) MSIX IRQ Handled = 0
igb2: TX(3) Packets sent = 0
igb2: Queue(0) rdh = 246, rdt = 245
igb2: RX(0) Packets received = 13727912438
igb2: RX(0) Split Packets = 0
igb2: RX(0) Byte count = 10677902707967
igb2: RX(0) MSIX IRQ Handled = 8730210616
igb2: RX(0) LRO Queued= 0
igb2: RX(0) LRO Flushed= 0
igb2: Queue(1) rdh = 51, rdt = 50
igb2: RX(1) Packets received = 13676394291
igb2: RX(1) Split Packets = 0
igb2: RX(1) Byte count = 10622417591526
igb2: RX(1) MSIX IRQ Handled = 8706454649
igb2: RX(1) LRO Queued= 0
igb2: RX(1) LRO Flushed= 0
igb2: Queue(2) rdh = 194, rdt = 193
igb2: RX(2) Packets received = 13687518658
igb2: RX(2) Split Packets = 0
igb2: RX(2) Byte count = 10608204213317
igb2: RX(2) MSIX IRQ Handled = 8716002420
igb2: RX(2) LRO Queued= 0
igb2: RX(2) LRO Flushed= 0
igb2: Queue(3) rdh = 131, rdt = 130
igb2: RX(3) Packets received = 13660077699
igb2: RX(3) Split Packets = 0
igb2: RX(3) Byte count = 10588060857368
igb2: RX(3) MSIX IRQ Handled = 8692141405
igb2: RX(3) LRO Queued= 0
igb2: RX(3) LRO Flushed= 0
igb2: LINK MSIX IRQ Handled = 32
igb2: Mbuf defrag failed = 0
igb2: Std mbuf header failed = 0
igb2: Std mbuf packet failed = 0
igb2: Driver dropped packets = 0
```
We have changed the cable between bridge and borfer1...but haven`t got any results


----------

