[SOLVED] How many PCIe lanes to use to connect computers together for a compute cluster?

[joshua123033]

If you are connecting computers over PCIe to become a compute cluster, how many PCIe lanes are enough? Is it x4? x16? Or do you need more (like 2 x16 slots)? Or does it not matter?

 

[kanewolf]

Generally, computers aren't connected into clusters via PCIe because the length of PCIe cables is limited. You can convert to fiber, but direct PCIe is not used. Clusters are connected via ethernet, or infiband. Most 10GE and infiband cards are PCIe x8. 25GE and even 100GE are starting to enter into high performance computing.

 

[joshua123033]

Generally, computers aren't connected into clusters via PCIe because the length of PCIe cables is limited. You can convert to fiber, but direct PCIe is not used. Clusters are connected via ethernet, or infiband. Most 10GE and infiband cards are PCIe x8. 25GE and even 100GE are starting to enter into high performance computing.

So a x16 lanes is overkill and x8 for compute clusters is enough?
Or is x16 lanes for other reasons (e.g. could it be for bandwidth purposes). Please explain.

 

[kanewolf]

The number of PCIe lanes required will depend on the type of interface used. A 10GE or 40GE card is going to be a PCIe x8 gen 3 card. A 100GE card is going to be an a PCIe x16 gen 3 card. Clusters are not the optimum architecture for a high I/O problem. Clusters work best for a problem that can have smaller data streams that require a large amount of computation.
Google Raspberry PI clusters. You can see that a functional compute cluster can be created with any level of hardware and connectivity.

 

[joshua123033]

The number of PCIe lanes required will depend on the type of interface used. A 10GE or 40GE card is going to be a PCIe x8 gen 3 card. A 100GE card is going to be an a PCIe x16 gen 3 card. Clusters are not the optimum architecture for a high I/O problem. Clusters work best for a problem that can have smaller data streams that require a large amount of computation.
Google Raspberry PI clusters. You can see that a functional compute cluster can be created with any level of hardware and connectivity.

Could you give examples about "problems that can have smaller data streams that require a large amount of computation.", and how "Clusters are not the optimum architecture for a high I/O problem." please?
Also, (I really don't want to sound like an annoying person who just wants to game but) could you game on it? I mean, couldn't you use the compute clusters as coprocessors (which they are), use them together in a VM, and then game on them. I mean they have low latency (if you use PCIe-over-cable) and they have lots of bandwidth (PCIe x16 if you choose). Plus, I've researched that the reason why you can't game on Raspberry Pi clusters is because of latency, but with mini-PCIe, the low latency allows the individual computers to act as coprocessors.
My whole goal was to buy these little SBCs from Gateworks that had 1 mini-PCIe slot and then use them together in a cluster for a small PC setup (http://www.gateworks.com/product/item/newport-gw6100-single-board-computer).

 

[kanewolf]

A classic "cluster" problem is SETI @ Home. Discrete work units of radio data are distributed to thousands of home computers. That is a great example of a problem that has a small data stream.
Can you use a cluster for gaming? No. Why? Because the software wasn't written for a cluster. That is the second critical part to a cluster, you have to have software that was written for distributed processing (cluster). Gaming software requires all the resources in a single OS instance. Each node in a cluster is a separate OS. You might be able to get a game SERVER to be cluster aware so that as more users connect to the "server" (really server cluster), that the new users are uniformly distributed across all nodes.
The link you provided, is basically the same as a Raspberry PI. The reason you can't game on a Raspberry PI cluster isn't just the latency, it is the fact that it isn't a single OS instance and the software is not written to be distributed.

 

[joshua123033]

A classic "cluster" problem is SETI @ Home. Discrete work units of radio data are distributed to thousands of home computers. That is a great example of a problem that has a small data stream.
Can you use a cluster for gaming? No. Why? Because the software wasn't written for a cluster. That is the second critical part to a cluster, you have to have software that was written for distributed processing (cluster). Gaming software requires all the resources in a single OS instance. Each node in a cluster is a separate OS. You might be able to get a game SERVER to be cluster aware so that as more users connect to the "server" (really server cluster), that the new users are uniformly distributed across all nodes.
The link you provided, is basically the same as a Raspberry PI. The reason you can't game on a Raspberry PI cluster isn't just the latency, it is the fact that it isn't a single OS instance and the software is not written to be distributed.

Thanks man for all your answers and for putting up with all the explanations!