# FreeBSD 8.3 Intel E5-2620 + intel X520 DA2. How to increase productivity?



## allan_sundry (Oct 18, 2012)

Hi!

I have a server:


```
CPU: 2x Intel (R) Xeon (R) CPU E5-2620 0@2.00GHz (2000.01-MHz K8-class CPU) (HT off)
RAM: 8GB
NIC: Intel Server Adapter X520-DA2 (2 link x10G to swith)
FreeBSD 8.3-RELEASE-p4
```

I did a little tuning:

```
# cat /boot/loader.conf 
...
hw.ixgbe.rxd=4096
hw.ixgbe.txd=4096
...

# cat /etc/sysctl.conf 
...
# NMBClusters (default 25600)
kern.ipc.nmbclusters=512000
# JumboClusters (default 12800)
kern.ipc.nmbjumbop=256000
# Intel recomend increase (default 1000)
hw.intr_storm_threshold=9000
# IXGB
dev.ix.0.rx_processing_limit=4096
dev.ix.1.rx_processing_limit=4096
...
```

Each user interface Intel X520-DA2 provides 8 queues:

```
sysctl -a | grep ix.0           
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.5
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10fb subvendor=0x8086 subdevice=0x0003 class=0x020000
dev.ix.0.%parent: pci6
dev.ix.0.fc: 3
dev.ix.0.advertise_gig: 0
dev.ix.0.enable_aim: 1
dev.ix.0.advertise_speed: 0
dev.ix.0.rx_processing_limit: 4096
dev.ix.0.dropped: 0
dev.ix.0.mbuf_defrag_failed: 0
dev.ix.0.no_tx_dma_setup: 0
dev.ix.0.watchdog_events: 0
dev.ix.0.tso_tx: 25255
dev.ix.0.link_irq: 6
dev.ix.0.queue0.interrupt_rate: 250000
dev.ix.0.queue0.txd_head: 2148
dev.ix.0.queue0.txd_tail: 2148
dev.ix.0.queue0.no_desc_avail: 0
dev.ix.0.queue0.tx_packets: 2548424586
dev.ix.0.queue0.rxd_head: 193
dev.ix.0.queue0.rxd_tail: 192
dev.ix.0.queue0.rx_packets: 2542892969
dev.ix.0.queue0.rx_bytes: 92154394802
dev.ix.0.queue0.lro_queued: 0
dev.ix.0.queue0.lro_flushed: 0
dev.ix.0.queue1.interrupt_rate: 250000
dev.ix.0.queue1.txd_head: 2645
dev.ix.0.queue1.txd_tail: 2645
dev.ix.0.queue1.no_desc_avail: 434
dev.ix.0.queue1.tx_packets: 2538223612
dev.ix.0.queue1.rxd_head: 824
dev.ix.0.queue1.rxd_tail: 823
dev.ix.0.queue1.rx_packets: 2520184131
dev.ix.0.queue1.rx_bytes: 76553875238
dev.ix.0.queue1.lro_queued: 0
dev.ix.0.queue1.lro_flushed: 0
dev.ix.0.queue2.interrupt_rate: 1000000
dev.ix.0.queue2.txd_head: 3042
dev.ix.0.queue2.txd_tail: 3044
dev.ix.0.queue2.no_desc_avail: 0
dev.ix.0.queue2.tx_packets: 2521877482
dev.ix.0.queue2.rxd_head: 666
dev.ix.0.queue2.rxd_tail: 662
dev.ix.0.queue2.rx_packets: 2497515347
dev.ix.0.queue2.rx_bytes: 68766948284
dev.ix.0.queue2.lro_queued: 0
dev.ix.0.queue2.lro_flushed: 0
dev.ix.0.queue3.interrupt_rate: 38461
dev.ix.0.queue3.txd_head: 1847
dev.ix.0.queue3.txd_tail: 1847
dev.ix.0.queue3.no_desc_avail: 21
dev.ix.0.queue3.tx_packets: 2511792679
dev.ix.0.queue3.rxd_head: 354
dev.ix.0.queue3.rxd_tail: 353
dev.ix.0.queue3.rx_packets: 2484006427
dev.ix.0.queue3.rx_bytes: 72600953889
dev.ix.0.queue3.lro_queued: 0
dev.ix.0.queue3.lro_flushed: 0
dev.ix.0.queue4.interrupt_rate: 22727
dev.ix.0.queue4.txd_head: 1199
dev.ix.0.queue4.txd_tail: 1199
dev.ix.0.queue4.no_desc_avail: 0
dev.ix.0.queue4.tx_packets: 2503894689
dev.ix.0.queue4.rxd_head: 632
dev.ix.0.queue4.rxd_tail: 628
dev.ix.0.queue4.rx_packets: 2494369894
dev.ix.0.queue4.rx_bytes: 68695300417
dev.ix.0.queue4.lro_queued: 0
dev.ix.0.queue4.lro_flushed: 0
dev.ix.0.queue5.interrupt_rate: 21276
dev.ix.0.queue5.txd_head: 976
dev.ix.0.queue5.txd_tail: 976
dev.ix.0.queue5.no_desc_avail: 0
dev.ix.0.queue5.tx_packets: 2492246991
dev.ix.0.queue5.rxd_head: 682
dev.ix.0.queue5.rxd_tail: 681
dev.ix.0.queue5.rx_packets: 2485693582
dev.ix.0.queue5.rx_bytes: 74135189471
dev.ix.0.queue5.lro_queued: 0
dev.ix.0.queue5.lro_flushed: 0
dev.ix.0.queue6.interrupt_rate: 1000000
dev.ix.0.queue6.txd_head: 3477
dev.ix.0.queue6.txd_tail: 3477
dev.ix.0.queue6.no_desc_avail: 0
dev.ix.0.queue6.tx_packets: 2521664630
dev.ix.0.queue6.rxd_head: 369
dev.ix.0.queue6.rxd_tail: 368
dev.ix.0.queue6.rx_packets: 2502188306
dev.ix.0.queue6.rx_bytes: 71013517996
dev.ix.0.queue6.lro_queued: 0
dev.ix.0.queue6.lro_flushed: 0
dev.ix.0.queue7.interrupt_rate: 100000
dev.ix.0.queue7.txd_head: 2415
dev.ix.0.queue7.txd_tail: 2419
dev.ix.0.queue7.no_desc_avail: 0
dev.ix.0.queue7.tx_packets: 2505236977
dev.ix.0.queue7.rxd_head: 30
dev.ix.0.queue7.rxd_tail: 29
dev.ix.0.queue7.rx_packets: 2486859634
dev.ix.0.queue7.rx_bytes: 71129476131
dev.ix.0.queue7.lro_queued: 0
dev.ix.0.queue7.lro_flushed: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.local_faults: 16
dev.ix.0.mac_stats.remote_faults: 6
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.link_xon_txd: 0
dev.ix.0.mac_stats.link_xon_rcvd: 0
dev.ix.0.mac_stats.link_xoff_txd: 0
dev.ix.0.mac_stats.link_xoff_rcvd: 0
dev.ix.0.mac_stats.total_octets_rcvd: 15347000838325
dev.ix.0.mac_stats.good_octets_rcvd: 15346995346277
dev.ix.0.mac_stats.total_pkts_rcvd: 20033069037
dev.ix.0.mac_stats.good_pkts_rcvd: 20013548375
dev.ix.0.mac_stats.mcast_pkts_rcvd: 53207
dev.ix.0.mac_stats.bcast_pkts_rcvd: 810141
dev.ix.0.mac_stats.rx_frames_64: 1864192202
dev.ix.0.mac_stats.rx_frames_65_127: 6904972743
dev.ix.0.mac_stats.rx_frames_128_255: 726915375
dev.ix.0.mac_stats.rx_frames_256_511: 415966891
dev.ix.0.mac_stats.rx_frames_512_1023: 502229455
dev.ix.0.mac_stats.rx_frames_1024_1522: 9618756620
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.checksum_errs: 270379702
dev.ix.0.mac_stats.good_octets_txd: 16000191922608
dev.ix.0.mac_stats.total_pkts_txd: 20143207467
dev.ix.0.mac_stats.good_pkts_txd: 20143207467
dev.ix.0.mac_stats.bcast_pkts_txd: 735
dev.ix.0.mac_stats.mcast_pkts_txd: 19718
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.tx_frames_64: 3444758841
dev.ix.0.mac_stats.tx_frames_65_127: 4981392057
dev.ix.0.mac_stats.tx_frames_128_255: 719059016
dev.ix.0.mac_stats.tx_frames_256_511: 424448807
dev.ix.0.mac_stats.tx_frames_512_1023: 492128975
dev.ix.0.mac_stats.tx_frames_1024_1522: 10081419771
dev.ix.0.mac_stats.fc_crc: 0
dev.ix.0.mac_stats.fc_last: 0
dev.ix.0.mac_stats.fc_drpd: 0
dev.ix.0.mac_stats.fc_pkts_rcvd: 0
dev.ix.0.mac_stats.fc_pkts_txd: 0
dev.ix.0.mac_stats.fc_dword_rcvd: 0
dev.ix.0.mac_stats.fc_dword_txd: 0
```

The server is used as a border router - 2 full viev.

Each Intel Xeon E5-2620 has 6 cores - a total of 12 cores, Intel X650 DA2 with two interfaces 16 queues. As a result, part of the cores remain idle:

```
# # top -aSP

last pid: 30435;  load averages:  2.11,  1.92,  2.00                                                                                                                                                           up 1+07:32:26  13:50:08
57 processes:  2 running, 53 sleeping, 1 stopped, 1 waiting
CPU 0:   0.0% user,  0.0% nice,  1.4% system, 21.7% interrupt, 76.8% idle
CPU 1:   0.0% user,  0.0% nice,  1.4% system, 20.3% interrupt, 78.3% idle
CPU 2:   0.0% user,  0.0% nice,  1.4% system, 11.6% interrupt, 87.0% idle
CPU 3:   0.0% user,  0.0% nice,  2.9% system, 11.6% interrupt, 85.5% idle
CPU 4:   0.0% user,  0.0% nice,  0.0% system, 10.1% interrupt, 89.9% idle
CPU 5:   1.4% user,  0.0% nice,  0.0% system, 18.8% interrupt, 79.7% idle
CPU 6:   0.0% user,  0.0% nice,  4.3% system, 18.8% interrupt, 76.8% idle
CPU 7:   0.0% user,  0.0% nice,  1.4% system, 26.1% interrupt, 72.5% idle
CPU 8:   0.0% user,  0.0% nice,  1.4% system,  0.0% interrupt, 98.6% idle
CPU 9:   0.0% user,  0.0% nice,  1.4% system,  0.0% interrupt, 98.6% idle
CPU 10:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 11:  0.0% user,  0.0% nice,  1.4% system,  0.0% interrupt, 98.6% idle
Mem: 513M Active, 15M Inact, 587M Wired, 84K Cache, 76M Buf, 6732M Free
Swap: 

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root       12 171 ki31     0K   192K CPU0    0 313.6H 1052.25% [idle]
   12 root       62 -48    -     0K   992K WAIT    0  47.5H 163.87% [intr]
    0 root       43 -68    0     0K   672K -      10 969:11  7.81% [kernel]
  671 root        1  46    0   305M   294M select  8  35:34  4.20% /usr/local/sbin/bgpd -d -A 127.0.0.1
  661 root        1  44    0   156M   144M select  7  14:08  0.10% /usr/local/sbin/zebra -d -A 127.0.0.1

...
```

How I can configure the binding queues ix0 only CPU1 cores, and queues ix1 only CPU2 cores (eg reducing the number of queues to 6 on each interface)?


----------

