How connect Infiniband switch to ethernet(SFP+) ?

[atamgp]

Hi All,

I have 2 layers of clusters.
Layer 1 has front-end clusters (multiple) which use gigabit.
Layer 2 use a distributed file-system so support all Layer 1 clusters.

The Layer 2 nodes need to be very fast connected to each other. For that i will use 2 x Infiniband (10GB/s) for each node to 2 Infiniband switches (HA).

My Layer 1 server are connected using cat6 gigabit (up to 4x) to a L2 switch (per cluster) which has 4 x SFP+ options for 10GB.
e.g. this one


Layer 1 needs to be connected to my internet facing firewall -> no problem.

Problem: How do i connect my Layer 2 (infiniband) to all Layer 1 switches (SFP+)?

Thanks a lot,
Atam

 

[bill001g]

You will need a device that bridges the ethernet to infiniband. The switches you link do not have that ability but IF your infiniband switches have 10g ethernet ports then they will do that. They do make standalone bridge devices but it is more common to buy this as a option in your infiniband switch since it lets it use the backplane rather than having to buy lots of interfaces to cross connect all this stuff.

 

[atamgp]

hmmm.... that sounds like $$$ involved.

My current switches only have IB ports (topspin 120 = cisco).

Can i build a custom bridge e.g.
- quadcore cpu
-4Gb ram
-2 X 10GB connections per IB switch
-1 x 10g ethernet per cluster switch
- running pfsense???

Or some other solution around Euro 500...

Thanks in advance.

 

[bill001g]

get very complex when you want to retain your redundancy. I would suspect you could build your own but I have never done it. Most the time when I get involved with stuff like this performance is key and money doesn't matter so they just buy the proper switches.

I know the last one we put in the servers had 2 connection 1 to the ethernet to talk to users and a second to the infiniband to access the storage. In this case the users did not directly access the data they issued transactions to the server.

You may want to try to post this to the server forum maybe someone else has a better soltuion.

 

[atamgp]

ok. I think that the best way is to go with IB adapters on Layer 1 to connect to switches and beneath (layer 2),
and ethernet adapters to connect to internet etc.

Thanks

 

[jmargolis]

I actually work in the IB business, and I can tell you that our bridge (4036E) brand new isn't cheap for a small home use cluster. But, I've had customers that have built their own `routers` using Linux using stuff like iptables. You can drop me a note if you want to discuss further.

 

[atamgp]

Hi jmarglois,

I thought I send you a note, perhaps it went wrong...

Anyway, I'm very interested in the solution your customers went with.
Perhaps also some information regarding how satisfied they are and what performance/stability they reach.

Thanks a lot.