# Only two CPUs in use



## personalraul (Dec 14, 2012)

Hi all 

We are using Dummynet on FreeBSD to emulate networks delay. We are running a high traffic and we have detected some problems. Even with any delay, packet loss etc in the pipes we reach a high use in our cpus.


We have this HW:

```
sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: i386
hw.model: Dual-Core AMD Opteron(tm) Processor 2218
hw.ncpu: 4
hw.machine_arch: i386
```
When we run our traffic, we can see this info in top :

```
top -aSCHIP

last pid: 13150;  load averages:  0.00,  0.00,  0.00                                                                                                            up 3+17:39:16  10:50:25
80 processes:  7 running, 57 sleeping, 16 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 80.1% interrupt, 19.9% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system, 84.0% interrupt, 16.0% idle
Mem: 14M Active, 145M Inact, 135M Wired, 112M Buf, 2961M Free
Swap: 16G Total, 16G Free

  PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME    CPU COMMAND
   13 root     171 ki31     0K     8K CPU1   1  88.5H 100.00% [idle: cpu1]
   12 root     171 ki31     0K     8K RUN    2  87.7H 100.00% [idle: cpu2]
   38 root     -68    -     0K     8K CPU3   3  98:36 86.47% [irq258: bce0]
   39 root     -68    -     0K     8K CPU0   0  55:41 83.01% [irq259: bce1]
   14 root     171 ki31     0K     8K RUN    0  86.9H 22.90% [idle: cpu0]
   11 root     171 ki31     0K     8K CPU3   3  87.7H 18.95% [idle: cpu3]

top -aSP
last pid: 13170;  load averages:  0.00,  0.00,  0.00                                                                                                            up 3+17:45:06  10:56:15
80 processes:  8 running, 57 sleeping, 15 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 73.8% interrupt, 26.2% idle
CPU 1:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 2:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system, 74.6% interrupt, 25.4% idle
Mem: 14M Active, 145M Inact, 135M Wired, 112M Buf, 2961M Free
Swap: 16G Total, 16G Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   13 root        1 171 ki31     0K     8K RUN    1  88.6H 100.00% [idle: cpu1]
   12 root        1 171 ki31     0K     8K RUN    2  87.8H 100.00% [idle: cpu2]
   38 root        1 -68    -     0K     8K CPU3   3 103:13 83.59% [irq258: bce0]
   39 root        1 -68    -     0K     8K CPU0   0  60:07 79.44% [irq259: bce1]
   14 root        1 171 ki31     0K     8K RUN    0  86.9H 27.00% [idle: cpu0]
   11 root        1 171 ki31     0K     8K RUN    3  87.7H 22.02% [idle: cpu3]
```

Like you can see, we have 4 CPUs but we are using only 2. An these two cpus (cpu0 and cpu3) have a very high value for interrupt cpu level(84.0% interrupt).

We have tried with these commands, but situation ( only 2 cpu's in use) remains:

```
cpuset -g
pid -1 mask: 0, 1, 2, 3
cpuset -l 0-3 -s 1
cpuset -g
pid -1 mask: 0, 1, 2, 3
```

Another option is improve system behavior to avoid this high level of interruptions.



Any support will be wellcome:

1- How to force to use 4 cpu's
2- How to improve system behavior to reduce interruptions.

thanks in advances.


----------



## SirDice (Dec 14, 2012)

personalraul said:
			
		

> 2- How to improve system behavior to reduce interruptions.


Have a look at polling(4) and tuning(7).


----------



## phoenix (Dec 14, 2012)

Which NIC(s) are you using?  And how many in total?


----------



## personalraul (Dec 17, 2012)

Hi

Thanks SirDice and phoenix by your answer.

We will try with polling, I will inform you.


About NIC:

```
DUMMYNET06# ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:17:08:7e:67:16
        inet 10.42.84.184 netmask 0xffffff00 broadcast 10.42.84.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:17:08:7e:67:17
        media: Ethernet autoselect
        status: no carrier
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:19:bb:2e:57:fe
        inet 10.1.6.2 netmask 0xfffffffc broadcast 10.1.6.3
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
        ether 00:19:bb:2e:57:f2
        inet 10.2.6.2 netmask 0xfffffffc broadcast 10.2.6.3
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:cd:69:e3:81:11
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: bce1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 55
        member: bce0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
DUMMYNET06#
```


----------



## phoenix (Dec 17, 2012)

AFAIK, bce(4) doesn't support multiple queues.  Thus, 1 NIC == 1 queue == 1 CPU used to handle interrupts from that NIC.  Since you have 2 NICs, only 2 CPUs will be used.


----------



## personalraul (Dec 18, 2012)

Hi,

Ok, then with these NICs we can use only 2 CPUs.

Thanks for your answer

What about polling: could we use it with this NIC? Could we use it with frebsd FreeBSD 7.2? 

We are using this tutorial:

http://www.cyberciti.biz/faq/freebsd-device-polling-network-polling-tutorial/

Thanks in advance.


----------



## SirDice (Dec 18, 2012)

personalraul said:
			
		

> Could we use it with frebsd FreeBSD 7.2?


FreeBSD 7.2 is end-of-life since June 2010 and thus unsupported. Update to at least 7.4.


----------



## Rudy (Dec 21, 2012)

Wow, you are running i386 on an Opteron system.  Try amd64 -- maybe it will benchmark faster.

What is your output of:

```
sysctl net.isr
```
Would using using netisr help even though bce only has one queue?

```
cat net.isr.maxthreads=4 >> /boot/loader.conf
```

You could switch to your em ports, or drop the bce card and use an igb card.  Lots of things to test!


----------

