# Intel 82599 and 1 GbE RJ45 SFP modules



## Tsuroerusu (Apr 27, 2019)

I am trying to use an Intel 82599-based NIC (Supermicro's AOC-STGN-i2S) with StarTech's SFP1000TXST RJ45 SFP module.
If I do "ifconfig -m ix0" when the module is plugged in, it reports:


```
supported media:
                media autoselect
                media 1000baseT
```

Which seems logical given that, that is the speed of the module. My switch also indicates that the connection is at 1 GbE in speed.

However, no matter what I do, I keep getting "no carrier". Whether I configure it using rc.conf or use ifconfig manually, it keeps giving me "no carrier" and no pings get through to the gateway.

The strange thing is that, when I look in my dmesg, I do not get the "unsupported sfp" message. On their website, Intel states at least one MSA-compliant RJ45 SFP module from Avago, and this one should be similar to that, which squares with the ix driver not saying "unsupported sfp".

Also, in the specs for the controller on Intel's website, they state that the port speed can be either 10 GbE or 1 GbE.

The ix driver has been driving me up the fall in other contexts, but this is just bizarre to me. Does anybody have any ideas if this can be made to work?


----------



## Phishfry (Apr 28, 2019)

I had trouble getting some legacy 1GB gear interfaced with 10G gear. I think adding 'SFP to RJ45' adds trouble.
What you found with Intel board is common for them. I advise you use Intel branded SFP modules.
Even used they work better than clones. Some boards drivers check for SFP blob some don't.
The real problem you are facing is using 1GbE on a 10GbE network. I had to buy Cisco 1 GbE SFP modules for my SG300 and I found Finisar 1GbE SFP modules for my NetApp fiber switch. Note: Not SFP+ modules.
MGBSX1 << Cisco 1000-baseSX SFP module for my SG300 switch. They check for a blob.
FTLF8519P2BNL << Finisar modules seem to be very compatible on switches.

10GbE may be backwards compatible but I could not get 1GbE working over 10GbE. When I used 1 GbE modules on both ends it works. I use LC-LC OM3 multimode duplex cabling just like 10G cable.

One thing I did notice is some 10G drivers (chelsio) give more SFP details than others WRT `ifconfig -vv` and SFP details.

Does your downstrream switch not offer any SFP ports?


----------



## Tsuroerusu (Apr 28, 2019)

Phishfry said:


> What you found with Intel board is common for them. I advise you use Intel branded SFP modules.
> Even used they work better than clones. Some boards drivers check for SFP blob some don't.


Judging by what Intel wrote on their website, I do not think they actually make RJ45 SFP modules themselves:

They mention two RJ45 modules that are, supposedly, tested to work with the 82599. Those being Avago ABCU-5710RZ and Finisar FCLF8522P2BTL. From what I could gather from looking into those, they are not Intel-branded, and at least the Avago one is an MSA-compliant module. But yeah, it would be interesting to test using one of those two, however at this point, having had to deal with the ix driver, my gut tells me that since these StarTech modules are MSA-compliant, and the driver does not say "unsupported sfp", but it still goes "no carrier", I think the result for those would probably be the same.

This is the page I was looking at:








						SFP+ Modules and Cables Compatible with Intel® Ethernet Converged...
					

SFP+ optical module, SFP module, and direct attach cable requirements for the Intel® Ethernet Converged Network Adapter X520 Series.




					www.intel.com
				







Phishfry said:


> Does your downstrream switch not offer any SFP ports?


It does have 4 x 10 GbE SFP+ ports, but those are already in use for other purposes, and that was why I was hoping to be able to use these RJ45 SFP modules, because I do not actually need the 10 GbE speed for the purpose of these Intel NICs (Whereas I do need it for what it already does).

Basically, I originally had two of these Intel 82599 cards in my OPNsense firewalls, but since August 2018 they do not work with VLANs because Intel screwed up the driver, so that it goes "no carrier" if you create VLANs. So I installed Mellanox ConnectX-3 NICs in my firewalls, and then I was hoping to be able to simply re-use these Intel NICs for other stuff. One feature they have going for them is their ability to supply 126 virtual functions for SR-IOV, and that is useful to me.

At this point, I am tempted to just get a smaller SKU of the switch I use, and just hook-up these Intel NICs via DAC cables. That will probably be more economical as well given that if the Avago and Finisar modules turned out to yield the same result, I would have more door-stops than I already do, lol.


----------



## Phishfry (Apr 28, 2019)

Tsuroerusu said:


> So I installed Mellanox ConnectX-3 NICs in my firewalls,


I use them too. The single port cards are 4X PCIe which worked out good for some client machines with limited IO.
Finisar on those work good.
Have not tried the Mellanox SR-IOV features yet. Using Chelsio T420/422 cards with SR-IOV and bhyve.


Tsuroerusu said:


> and just hook-up these Intel NICs via DAC cables


I had more trouble with DAC cables than anything. They are really brand specific.
Transciever is built-in so really best for connecting 2 pieces of equipment of the same brand.
I have all different brand gear and they were a waste of my money.

Here are the Intel branded SFP+ modules that I use with my X520 boards.
https://www.ebay.com/itm/113699659049


----------



## Tsuroerusu (Apr 28, 2019)

Phishfry said:


> I use them too. The single port cards are 4X PCIe which worked out good for some client machines with limited IO.
> Finisar on those work good.
> Have not tried the Mellanox SR-IOV features yet. Using Chelsio T420/422 cards with SR-IOV and bhyve.


Thanks for the tip, I was not aware that they had a single-port card. The Mellanox ConnectX-3 seem to, unfortuantely, only be able to do 8 VFs per port, and I need a good deal more than that for my purpose, where I need a VF per customer (And I need VF capacity to handle  at least 100). I am hoping that once I find an extra switch that I can get the 82599 cards to work, because they can do 63 VFs per port.




Phishfry said:


> I had more trouble with DAC cables than anything. They are really brand specific.
> Transciever is built-in so really best for connecting 2 pieces of equipment of the same brand.
> I have all different brand gear and they were a waste of my money.
> 
> ...


I am generally a Supermicro guy, and so far their DAC cables have worked great for me with the switch that I use. They work perfectly too going from the Mellanox CX3 cards in my firewalls to the switch. Amusingly, my switch detected that the cables were really Ampherol and Foxconn cables, so it seems quite generic in Supermicro's case. I appreciate your advice on matching brands, which is probably what I will generally be doing in the future to avoid wasting more money.


----------



## Phishfry (Apr 28, 2019)

Here is the Connect-X 3 single port card. MCX311A.
https://www.ebay.com/itm/253568380352

I also wanted to add this nice deal. A Chelsio T5 quad adapter. NO backplate though...
I made a custom plate as mine needed riser mounting due to full height expansion card in 2U chassis.
https://www.ebay.com/itm/123731995533
My guess is these came out of some sort of network appliance. I see no backplates for sale anywhere.
$105USD is a good price.


----------

