# Mellanox MCX311A running at 10GbE.



## Phishfry (Dec 9, 2018)

Well I picked up a MCX311A card because I need a 4x PCIe 10-Gigabit Ethernet card for my firewall.
The card is so small it is amazing. Half the height of a Low Profile bracket. One SFP+ socket.
They are selling for $27 bucks used on ebay so I bought one.
I was happy to find out that it works fine at 10GbE with no kernel modifications necessary.

/etc/rc.conf
ifconfig_mlxen0="DHCP"

/boot/loader.conf
mlx4en_load="YES"


```
# ifconfig -v mlxen0
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:02:c9:a1:d0:80
    hwaddr 00:02:c9:a1:d0:80
    inet 192.168.1.122 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
```

This is using a Finnstar SFP+ module to a 2m fiber LC-LC om3 dualmode cable connected to my Aruba switch.
I am using the Aruba 10GbE ports as normal DHCP ports instead of uplinks for now. I have a Netapp 10G switch but need to learn it..
Will report back in the future with some speeds between FreeBSD 10GbE devices like Chelsio T420 and Intel X520-DA2.


----------



## puretone (Jan 23, 2019)

Always fun to see mention of my home country in geeky hardware specs. AMD uses a device core codename Aruba for one of their graphics cards series as well. I ought to use one of their switches on the actual island, hehehe
Any particular reason you opted for the much more expensive (albeit much muuuch cheaper on eBay) ConnectX-3 cards?
Surely you meant to say Finisar SFP+


----------



## Phishfry (Jan 23, 2019)

I wanted PCIe 3 bus. Max blast for a PCIe 3.0 4x slot. I have some machines with only 4X PCIe slot and one 10GBe is enough.
Yes those sound like the SFP modules. Amazing little lasers.


----------



## Phishfry (Jan 23, 2019)

Can any fiber cable experts tell me why short cables are not more prevalent?
Most are =>1Meter. I am so fed up I am considering trying to make my own.
I want a tidy rack.
Maybe short runs most people use twinax direct attach instead?
I have gear stacked right net to one another and need shorty multimode LC OC3 10G patch cables.<=.3M


----------



## pboehmer (Jan 23, 2019)

Not an expert, but I'd imagine because fiber was designed for long(er) haul connections.  With < 1m length, you could (should?) use 10g copper.  That being said, you may want to make sure you're not running too hot on your connections running a risk of burning out your SPFs.


----------



## inf3rno (Dec 23, 2020)

Phishfry said:


> Well I picked up a MCX311A card because I need a 4x PCIe 10-Gigabit Ethernet card for my firewall.
> The card is so small it is amazing. Half the height of a Low Profile bracket. One SFP+ socket.
> They are selling for $27 bucks used on ebay so I bought one.
> I was happy to find out that it works fine at 10GbE with no kernel modifications necessary.
> ...


What transfer speed did you measure with it? I can do only 5-6 Gbps with iperf3, no matter how I try. I have a DAC cable between the computers. It is weird, because it seems to depend on the CPU of the sending computer a lot. My weaker computer can send only 4.7Gbps my stronger something around 5.8. The CPU is 3.4 GHz, I wonder if that is not enough for 10Gbps data transfer or something else is wrong with these systems. Another idea that I bought fake cards, but I would not expect fake cards to work with the Mellanox driver at 5-6 Gbps speeds. So it is weird, but I have no experience with high speed ethernet to know for sure...


----------



## Phishfry (Dec 23, 2020)

I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
I was using a single socket LGA2011 board with a V3 cpu for testing..


----------



## inf3rno (Dec 24, 2020)

Phishfry said:


> I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
> I was using a single socket LGA2011 board with a V3 cpu for testing..


Thanks! It turned out that one of the motherboards has only x2 pcie2 lanes in the second x16 slot, which limits it speed to the theoretical 8Gbps. I guess in practice that means 6Gbps and that's why I don't have higher speeds. I'll buy a new motherboard and we'll see. In theory these cards can do 40Gbps too with a modified firmware.


----------



## inf3rno (Jan 1, 2021)

Phishfry said:


> I doubt you got fake cards. I do not remember the exact speeds I achieved. Somewhere around 8500 Mbps if I remember right.
> I was using a single socket LGA2011 board with a V3 cpu for testing..


I bought a new motherboard meanwhile with 4x pcie3 lanes. Now I have 9.55Gbps with iperf3. With FTP it is only about 2.5Gbps, but certainly not the Mellanox card is the bottleneck.


----------

