Nobody ever really mentions the underlying network connecting these nodes.
It's Intel's Omnipath (100 Gbps). Check the link in my post.
It sounds very much like they prioritized building this with existing, off-the-shelf tech. From Broadwell-E Xeons to P100 GPUs, it sounds like they're trying to keep it (relatively) cheap and simple. Even the ratio of 4x P100's with 2x Xeons is surely about avoiding the need for a PCIe switch (each of those Xeons has 40x PCIe 3.0 lanes, which it can use to host 2x P100s).