[SOLVED] Switch (Arista) And NIC (Mellanox) Compatibility Question

Feb 12, 2022
38
0
30
0
Hello,

My company plans on building a system with the following NIC: Mellanox MCX653106A-HDAT


...and I was wondering whether it would work with the following switch: Arista 7050SX-72Q

https://www.arista.com/assets/data/pdf/Datasheets/7050SX-128_64_Datasheet.pdf

https://pasteboard.co/XkIEngOWVrIA.png

Even though the switch is older (and can't handle the high throughput of the Mellanox), the plan is to eventually replace the Arista (but for now, just use the Mellanox at a lower throughput),

I assume the Arista and the Mellanox should be able to auto-negotiate with one another, but one of the Engineers on my team mentioned that QSFP+ (on the Arista) couldn't work with QSFP56 (on the Mellanox).

In case the Mellanox doesn't work with the Arista, could someone recommend the highest throughput NIC that can work with the Arista?

Any help would be greatly appreciated.

Thank you,
Nelson
 
Your engineer is very correct. He might be a good source to ask for a recommendation.

Things like QSFP+ and QSFP56 represent the method data is being send over the fiber. It is even more complex because there are more versions of QSFP and they make optics that goes different distances. Key is the optical models on both ends must match.

Been a long time since I dealt with stuff like this. You should be able to find a card that has a QSFP+ interface. Best if you get one that is not fixed so you can use different optic to match the switch. They make short and long haul optics.

How fast the card actually runs is another story though. Generally when you start talking about transferring data that fast the bottleneck is some place other than the network. I know most storage networks do not actually use ethernet because of how much overhead there is.
 

kanewolf

Titan
Moderator
Hello,

My company plans on building a system with the following NIC: Mellanox MCX653106A-HDAT


...and I was wondering whether it would work with the following switch: Arista 7050SX-72Q

https://www.arista.com/assets/data/pdf/Datasheets/7050SX-128_64_Datasheet.pdf

https://pasteboard.co/XkIEngOWVrIA.png

Even though the switch is older (and can't handle the high throughput of the Mellanox), the plan is to eventually replace the Arista (but for now, just use the Mellanox at a lower throughput),

I assume the Arista and the Mellanox should be able to auto-negotiate with one another, but one of the Engineers on my team mentioned that QSFP+ (on the Arista) couldn't work with QSFP56 (on the Mellanox).

In case the Mellanox doesn't work with the Arista, could someone recommend the highest throughput NIC that can work with the Arista?

Any help would be greatly appreciated.

Thank you,
Nelson
QSFP (40GE) and QSFP56(100GE) may not play together. NVIDIA says those cards will work at 40GE, but you may need to be sure you get the correct optics.
 
Feb 12, 2022
38
0
30
0
Your engineer is very correct. He might be a good source to ask for a recommendation.

Things like QSFP+ and QSFP56 represent the method data is being send over the fiber. It is even more complex because there are more versions of QSFP and they make optics that goes different distances. Key is the optical models on both ends must match.

Been a long time since I dealt with stuff like this. You should be able to find a card that has a QSFP+ interface. Best if you get one that is not fixed so you can use different optic to match the switch. They make short and long haul optics.

How fast the card actually runs is another story though. Generally when you start talking about transferring data that fast the bottleneck is some place other than the network. I know most storage networks do not actually use ethernet because of how much overhead there is.
Hello Bill,

A big thanks for the explanation.

I thought that the different SFP's were like PCIe, with 4.0 being faster than 3.0, but still being able to fit into 3.0.

I didn't know that different SFP's are really like completely different protocols/interfaces all together.

This now makes sense.

Thank you for the help and input,
Nelson
 
Feb 12, 2022
38
0
30
0
QSFP (40GE) and QSFP56(100GE) may not play together. NVIDIA says those cards will work at 40GE, but you may need to be sure you get the correct optics.
Hello KaneWolf,

Understood. But just to be certain, by same optics, you mean same SFP type, correct?

If so, I will now look for a NIC that supports QSFP+.

Thank you,
Nelson
 

kanewolf

Titan
Moderator
Hello KaneWolf,

Understood. But just to be certain, by same optics, you mean same SFP type, correct?

If so, I will now look for a NIC that supports QSFP+.

Thank you,
Nelson
Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.
 
Feb 12, 2022
38
0
30
0
Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.
Hello KaneWolf,

Understood.

As for 40GE using MPO optical cables, no I did not know that (and to be completely honest, don't know a whole lot about network equipment).

I'm starting to understand now thanks to you and Bill001g.

Thanks again for all your help,
Nelson
 
Feb 12, 2022
38
0
30
0
Meaning 40Gb optics for the QSFP56. In other situations I have encountered multi-speed devices like that, the optics determined the rate. This Melanox may be different, but it is something you should ask the vendor. It might be the case that you purchase either Infiniband or ethernet and the speed is software controlled. BUT optics can be expensive, especially single mode optics. So just be sure you get proper optics. Are you aware that 40GE uses MPO optical cables for multimode connectivity? LC optics are only on single mode.
Hello Kane Wolf (or anyone else),

I found two cards which should work, between the following two cards: St. Helens 200G4Q-XL

https://www.hotlavasystems.com/products_40gbe.html

...or: PE31640G4QI71 Server Adapter


...do you prefer/recommend a specific card?

They seem pretty much the same, but I might be missing something.

Thank you once again for all your help,
Nelson
 

ASK THE COMMUNITY