# 10Gb Ethernet PCI-E



## dndlnx (Nov 9, 2018)

What is the best PCI Express card for 10Gb networking on FreeBSD?

Must have a low-profile bracket, I should mention...


----------



## dndlnx (Nov 9, 2018)

Would Intel be a safe bet?


----------



## VladiBG (Nov 9, 2018)

http://www.si.freebsd.org/relnotes/CURRENT/hardware/support.html


----------



## Phishfry (Nov 9, 2018)

I wrote my opinion here yesterday:
https://forums.freebsd.org/threads/connectx-2-mhqh29c-xtr-in-freebsd-11-2.68169/post-406112

https://bsdrp.net/documentation/technical_docs/performance
Some nice graphs here.


----------



## Phishfry (Nov 10, 2018)

Well the SFP+ model I posted in my recomendation has the modules included.(For only 4 bucks more)
Here is one with no modules. (Note the custom backplate as well)
https://www.ebay.com/itm/132753046532

The reason I posted the one with modules is that you know the SFP+ modules work on Chelsio.
Whereas if you have to buy the modules separately you need to keep your eye on compatibility.
They are different brand to brand.
Also the modules included in the above auction have no details. They are probably 850nm as these are the most common.
Made for short distance compared to 1310 or 1550nm for long distance.
You can use whatever module is needed for the network. They do need to be compatible.
Intel SFP+ modules might not work on Chelsio.

There are also older Generation 3 Chelsios with the SFP+ embedded onboard. No module needed.
https://www.ebay.com/itm/401609416179

https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver


----------



## dndlnx (Nov 10, 2018)

This new 10gbe eludes me to be honest, but I dunno, I'm tempted to try an Intel solution for the "reputation".


----------



## Phishfry (Nov 10, 2018)

They have a good reputation on FreeBSD for gigabit ethernet. Not so much for 10G
From the bsdrouter project page.


> Intel seems to have problem for managing lot's of PPS (= IRQ).


I am not making this stuff up, just trying to help. There are many other references to their MSI-X problem.
You will have to find them. Sometimes they are just small annotations in code.

I come from pfSense so I fully understand the Intel first attitude. I live it and reject any board with anything but Intel gigabit interfaces.
This does not mean I ignore all the advice I read that states Intel 10Gbe is not worthy.

The very fact that 2 big storage houses use FreeBSD and Chelsio says it all. (EMC's Isilon and NetApp)

Just a FYI: There are many fake Intel x540's on ebay. If you have a good eye you can spot the inconsistencies.
That card from NewEgg is not sold from NewEgg and I would be wary.


----------



## dndlnx (Nov 10, 2018)

Appreciate your input, I'm giving the Chelsio ones a hard look.
10Gb Ethernet... it's a strange, new world.


----------



## Phishfry (Nov 10, 2018)

Why not consider 10G-BaseT or RJ45. Unless you have specific needs for fiber. The RJ45 models are more expensive on ebay because they are more in demand.
The ones I bought were in factory new box with original packing. I don't know why the listing calls them used. $110 is more than double the SFP+ model but the fact that I can make my own cables for twisted pairs makes it worth it to me. Long haul I would definatly buy SFP+. My runs are under 15 meters. You also have to consider how you are connecting them in your network.
The 10G-BaseT managed switches are quite expensive still.

One thing that you want to watch is heat. These cards are built for servers and need good airflow to keep them cool.

```
dev.cxgbe.1.temperature: 50
dev.cxgbe.0.temperature: 50
dev.t4nex.0.temperature: 66
```

I would like to see how much difference in heat there is between the T4 and T5 models. I don't have any T5's yet.


----------



## dndlnx (Nov 10, 2018)

I'd like a basic network that can transfer 10Gb between a file server, and maybe one other computer. But I might be getting ahead of myself, there's the implications of a 10Gbe network. Which would include a 10Gb router or switch of some kind?

As far as internet speeds and fiber, I don't think my current ISP could keep up. It'd be more for local transfers.


----------



## Phishfry (Nov 10, 2018)

My approach was to take my Bhyve server and to additionally task it as my 10G switch. I installed two Chelsio T420 in it and attached two other servers to it, both also using Chelsio 10G cards.. I used my Bhyve servers motherboard gigabit for LAN and set it up as a gateway with dnsmasq and I use a separate subnet to feed my two other servers. A storage machine and a build box.
I also have a T420 with SFP+ for connection to a Quanta serverboard that I am waiting on from ebay. It only has SFP+ so I had to do something different. Originally I had only planned on 10G-BaseT only network.
I still have not figured out how I will connect it. I am seriously looking at Twinax direct attach cabling instead of fiber.
Unsure if the Chelsio Twinax cable will work with the Quanta motherboard SFP+ (Intel) though...

That is the nice thing about RJ45. You don't have to worry about SFP+ module compatibility or expensive cabling.

I should also mention that the T4 Chelsio are only PCIe 2.x whereas the T5 series is PCIe 3.0.


----------



## DenisVS (Jan 21, 2022)

Recently I bought two ZyXEL XGN100F cards for my new server, but unfortunately there are nothing in output of ifconfig, dmesg and pciconf.
Eventually I found interesting topic How to install drivers for a NIC? (AQtion AQC107/XG-C100C), so I wanna try it.


----------

