# Impact of mixing PCIe 2.0 & 2.1 2-port Cu NICs in PCIe 3.0 motherboard Intel C600 Series with 2xXeon 8-Core CPU



## Sergei_Shablovsky (Apr 11, 2022)

Hi FreeBSD Gurus!

We have FreeBSD (12 but planning to update to 13) installed on a old (born in 2014) server:
PCIe 3.0 motherboard Intel C600 Series with 2xXeon 8-Core CPU
 with a 6 of PCIe v.3
4 of x8 slots (dedicated to CPU 1)
2 of x16 slots (dedicated to CPU 2)

Both CPU are:
Xeon E5-2670 8C/16T 2.60GHz 20MB 8.00GT/s 1600MHz 115W

How impact mixing the 2-head NICs:
Intel® I350-AM2 Powerwille based  - 1Gb/s - PCIe 2.1
ftp://ftp.kontron.com/Products/Accessories/EoL_Components/LAN_WLAN/DS_F5000-L004_Dual-LAN_D3035.pdf
Intel® 82576EB Kawela based NICs - 1Gb/s - PCIe 2.0


			https://sp.ts.fujitsu.com/dmsp/Publications/public/ds-py-lan-adapter-d2735-2.pdf
		


to overall *network performance* for *delay/jitter sensitive applications* like *video/voice streaming*?

Is that CPU loading increases because higher IRQs numbers from NICs, etc... ?

Tank You all for detailed answering!


----------



## Sergei_Shablovsky (May 1, 2022)

Any opinions?


----------



## Phishfry (May 1, 2022)

When you have a Dual CPU machine I feel it is best to group together functions.
All my 10G NIC's are on one CPU and all my NVMe are on the other CPU.
NUMA is how information is shared between CPU's. It is slower than QPI which is built into the CPU.
So my thought is if you can keep the tasks from jumping CPU you save latency.
For example a gmirror of NVMe should not use different CPU's.
NUMA is a slight performance bottleneck.
Likewise a LAGG across CPU's buses would not be desirable.

I can't talk to your interrupt problem.
Study your `vmstat -ai` output.

Here is my LAGG0 on my router.

```
[SNIP]
irq67: t5nex0:0a9                 257129          1
stray irq67                            0          0
irq68: t5nex0:1a0                 218233          0
stray irq68                            0          0
irq69: t5nex0:1a1                 245691          1
stray irq69                            0          0
irq70: t5nex0:1a2                 286330          1
stray irq70                            0          0
irq71: t5nex0:1a3                 239806          0
stray irq71                            0          0
irq72: t5nex0:1a4                 195039          0
stray irq72                            0          0
irq73: t5nex0:1a5                 186268          0
stray irq73                            0          0
irq74: t5nex0:1a6                 211110          0
stray irq74                            0          0
irq75: t5nex0:1a7                 222689          0
stray irq75                            0          0
irq76: t5nex0:1a8                 174816          0
stray irq76                            0          0
irq77: t5nex0:1a9                 153821          0
```

My motherboard ethernet port used as WAN:

```
irq183: em0:rxq1                 2565309          5
stray irq183                           0          0
```


----------



## Phishfry (May 1, 2022)

Sergei_Shablovsky said:


> How impact mixing the 2-head NICs:
> to overall *network performance* for *delay/jitter sensitive applications* like *video/voice streaming*?
> 
> Is that CPU loading increases because higher IRQs numbers from NICs, etc... ?


I am having a hard time with your question.

It seems from your title that you are concerned about 2 NIC's using different bus protocols.
That should not be an issue at all. Gigabit ethernet will not top out your bus. PCIe 2.0 or PCIe 2.1.
So mixing those cards should not be a problem at all.


----------



## Phishfry (May 1, 2022)

Sergei_Shablovsky said:


> How impact mixing the [different] 2-head NICs:
> to overall *network performance*


Lets just chop the question up.

Without knowing your network this is impossible. Do you have switches in your network?
Are you connecting to clients directly?

The older NIC will use more interrupts. It will also run hotter.
Nothing to affect performance.


----------

