[SOLVED] Switch (Arista) And NIC (Mellanox) Compatibility Question

[TryingToVirtualizeStuff]

Hello,

My company plans on building a system with the following NIC: Mellanox MCX653106A-HDAT

NVIDIA Mellanox: End-to-End Interconnect Networking Solutions

The leading supplier of end-to-end Ethernet and InfiniBand intelligent interconnect solutions and services.

store.nvidia.com

...and I was wondering whether it would work with the following switch: Arista 7050SX-72Q

https://www.arista.com/assets/data/pdf/Datasheets/7050SX-128_64_Datasheet.pdf

https://pasteboard.co/XkIEngOWVrIA.png

Even though the switch is older (and can't handle the high throughput of the Mellanox), the plan is to eventually replace the Arista (but for now, just use the Mellanox at a lower throughput),

I assume the Arista and the Mellanox should be able to auto-negotiate with one another, but one of the Engineers on my team mentioned that QSFP+ (on the Arista) couldn't work with QSFP56 (on the Mellanox).

In case the Mellanox doesn't work with the Arista, could someone recommend the highest throughput NIC that can work with the Arista?

Any help would be greatly appreciated.

Thank you,
Nelson

 

[bill001g]

Your engineer is very correct. He might be a good source to ask for a recommendation.

Things like QSFP+ and QSFP56 represent the method data is being send over the fiber. It is even more complex because there are more versions of QSFP and they make optics that goes different distances. Key is the optical models on both ends must match.

Been a long time since I dealt with stuff like this. You should be able to find a card that has a QSFP+ interface. Best if you get one that is not fixed so you can use different optic to match the switch. They make short and long haul optics.

How fast the card actually runs is another story though. Generally when you start talking about transferring data that fast the bottleneck is some place other than the network. I know most storage networks do not actually use ethernet because of how much overhead there is.

 

[kanewolf]

Hello,

My company plans on building a system with the following NIC: Mellanox MCX653106A-HDAT

NVIDIA Mellanox: End-to-End Interconnect Networking Solutions

The leading supplier of end-to-end Ethernet and InfiniBand intelligent interconnect solutions and services.

store.nvidia.com

...and I was wondering whether it would work with the following switch: Arista 7050SX-72Q

https://www.arista.com/assets/data/pdf/Datasheets/7050SX-128_64_Datasheet.pdf

https://pasteboard.co/XkIEngOWVrIA.png

Even though the switch is older (and can't handle the high throughput of the Mellanox), the plan is to eventually replace the Arista (but for now, just use the Mellanox at a lower throughput),

I assume the Arista and the Mellanox should be able to auto-negotiate with one another, but one of the Engineers on my team mentioned that QSFP+ (on the Arista) couldn't work with QSFP56 (on the Mellanox).

In case the Mellanox doesn't work with the Arista, could someone recommend the highest throughput NIC that can work with the Arista?

Any help would be greatly appreciated.

Thank you,
Nelson

QSFP (40GE) and QSFP56(100GE) may not play together. NVIDIA says those cards will work at 40GE, but you may need to be sure you get the correct optics.

 

[TryingToVirtualizeStuff]

Your engineer is very correct. He might be a good source to ask for a recommendation.

Things like QSFP+ and QSFP56 represent the method data is being send over the fiber. It is even more complex because there are more versions of QSFP and they make optics that goes different distances. Key is the optical models on both ends must match.

Been a long time since I dealt with stuff like this. You should be able to find a card that has a QSFP+ interface. Best if you get one that is not fixed so you can use different optic to match the switch. They make short and long haul optics.

How fast the card actually runs is another story though. Generally when you start talking about transferring data that fast the bottleneck is some place other than the network. I know most storage networks do not actually use ethernet because of how much overhead there is.

Hello Bill,

A big thanks for the explanation.

I thought that the different SFP's were like PCIe, with 4.0 being faster than 3.0, but still being able to fit into 3.0.

I didn't know that different SFP's are really like completely different protocols/interfaces all together.

This now makes sense.

Thank you for the help and input,
Nelson

 

[TryingToVirtualizeStuff]

QSFP (40GE) and QSFP56(100GE) may not play together. NVIDIA says those cards will work at 40GE, but you may need to be sure you get the correct optics.

Hello KaneWolf,

Understood. But just to be certain, by same optics, you mean same SFP type, correct?

If so, I will now look for a NIC that supports QSFP+.

Thank you,
Nelson

 

[kanewolf]

Hello KaneWolf,

Understood. But just to be certain, by same optics, you mean same SFP type, correct?

If so, I will now look for a NIC that supports QSFP+.

Thank you,
Nelson

Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.

 

[TryingToVirtualizeStuff]

Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.

Hello KaneWolf,

Understood.

As for 40GE using MPO optical cables, no I did not know that (and to be completely honest, don't know a whole lot about network equipment).

I'm starting to understand now thanks to you and Bill001g.

Thanks again for all your help,
Nelson

 

[TryingToVirtualizeStuff]

Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.

Hello Kane Wolf (or anyone else),

I found two cards which should work, between the following two cards: St. Helens 200G4Q-XL

https://www.hotlavasystems.com/products_40gbe.html

...or: PE31640G4QI71 Server Adapter

Silicom Ltd. | 40G Networking Server Adapter PE31640G4QI71

Silicom’s Quad 40G Networking Server Adapter Ethernet PCI Express are designed for Servers and high-end appliances. Silicom's Quad 40Gigabit offer simple

www.silicom-usa.com

...do you prefer/recommend a specific card?

They seem pretty much the same, but I might be missing something.

Thank you once again for all your help,
Nelson

 

[kanewolf]

...do you prefer/recommend a specific card?

I will always pick a genuine Intel card first. The XL710 for example -- https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ethernet-xl710-brief.pdf