Latency. HTT latency is 33-40% of the FSB. that is why the new DARPA benchmarks focus on that. Red Storm using 130nm Clawhammer 146's bested Blue Gene L in two benchmarks because of latency.
"The two first-place benchmarks measure the efficiency of keeping track of data (called random access memory), and of communicating data between processors. This is the equivalent of how well a good basketball team works its offense, rapidly passing the ball to score against an opponent."
"More technically, Red Storm posted 1.8 TB/sec (1.8 trillion bytes per second) on one HPCC test: an interconnect bandwidth challenge called PTRANS, for parallel matrix transpose. This test, requiring repeated "reads," "stores," and communications among processors, is a measure of the total communication capacity of the internal interconnect. Sandia's achievement in this category represents 40 times more communications power per teraflop (trillion floating point operations per second) than the PTRANS result posted by IBM's Blue Gene system that has more than 10 times as many processors.
"Red Storm is the first computer to surpass the 1 terabyte-per-second (1 TB/sec) performance mark measuring communications among processors -- a measure that indicates the capacity of the network to communicate when dealing with the most complex situations.
"The "random access" benchmark checks performance in moving individual data rather than large arrays of data. Moving individual data quickly and well means that the computer can handle chaotic situations efficiently.
"Red Storm also did very well in categories it did not win, finishing second in the world behind Blue Gene in fft ("Fast Fourier Transform," a method of transforming data into frequencies or logarithmic forms easier to work with); and third behind Purple and Blue Gene in the "streams" category (total memory bandwidth measurement). Higher memory bandwidth helps prevent processors from being starved for data."
http://www.physorg.com/news62939660.html
When are supercomputers really super?
Interconnect speed is key to high-performance computing
http://www.gcn.com/print/25_5/40021-1.html A some what long article but lays most of the issues out in understandable terms.