"Ethernet latency: 350"
Ethernet isn't a flat 350µs latency. They must comparing a specific datagram/frame/packet size. I get a 0-1µs ping to my wife's computer over my gigabit switch.
In 1µs at gigabit speeds, you would've transfered 1000 bits or 125 bytes. Not very useful, but large enough for a 64byte ping packet.
Infinaband is awesome though. It's actually cheaper than 10gigabit networking, but it does have a VERY short limit on cable length. For ~$1500, you can get a new IB switch that could link 3 computers together with 40gb/s but a 12 meter max cable length and the cables are expensive. Each IB card costs about $1200, but a decent 10gb nic costs about the same. although, a fiber 10gb nic has a range of ~ 100xs that of IB.
But yes, what amno said. Locality of the data is very important since higher latency = slower computing. A multi-socket computer will have latency measured in nano-micro seconds while a "linked" computer would be in the micro-millisecond range.
Also, with heavy computing, the bottleneck really is how fast your CPUs are. It is better to pack more CPUs in the same computer, but after a while, the complexity of a large computer starts to become more of a problem than a blessing. It's probably best to have dual socket computers all network together via a very high-speed LAN.