To PLX PEX Or Not?

[DrGreer]

I've been grappling with this question now for days but still haven't come up with an answer. I've been researching buying the highest-end x99 motherboard I can find and the choice is very difficult. My CPU is a Broadwell-E with 40 lanes. I'm buying a high-end graphics card. I'm either going with two 1080s or a Titan X Pascal where in the future I would probably throw in a second Titan X.

Some of the high-end (And expensive of course) gaming motherboards I've looked at have this give and take that comes along with them. Throw anything in an m2 or u2 slot and a PCI slot or two will degrade or some other feature will be disabled. This give and take sometimes even creeps into the RAM slots and SATA connections. So what's to be done about that?

PLX to the rescue! Or so it would seem at first. There are some really nice boards that feature no give/take where you wind up with x16 across most if not all of the PCIe slots without having to sacrifice the usage of any component on the board; however, there are some threads out there where people claim that the use of PLX (Or PLX PEX, same thing I believe) creates added latency.

Now these PLX chips are on some really expensive high-end workstation and gaming boards where you really need high performance and stability. These boards have great reviews but there's still the latency question and the problem is I can't find any benchmarks to confirm or refute this claim. If it's true, I'd of course like to know exactly what it's going to affect and by how much. After all, these are expensive decisions and a Titan X Pascal is no laughing matter (Until you've got it screaming through computations).

My use-case is mainly running machine learning algorithms (Theano, Caffe, DL4J, Tensorflow) and VR game development. I've also read that SLI is no good for these machine learning libraries (A shame) but that they can make use of multiple GPUs in straight PCIe so the latency added by a PLX/PEX chip is of great importance. Do the benefits of PLX PEX outweigh the downside? We need some measure here.

Here are the cards I've narrowed my decision space down to:

• 
  **Asus X99-E-10G WS
  MSI X99A GODLIKE GAMING CARBON
  Asus Rampage V Edition 10
  *Gigabyte GA-X99-Designare EX

The * symbol denotes how many PLX chips are on board. Now I do have a quote from another thread where I was comparing these boards but it's just a quote and doesn't have any empirical data to go along with it:

This is from https://rog.asus.com/forum/showthread.php?81623-Which-ASUS-boards-have-the-PLX-chip-for-16x16-GPU-s on 12-30-15...

" PLX" is a bit of a misnomer, PLX Technologies is (was) a manufacturer of circuit components like PCIe switches and PCIe bridges and PCIe multiplexers. What "PLX" or "PLX chip" means to most people is a chip which splices a single physical PCIe slot (or lane, or group of lanes) into more logical PCIe slots or lanes.

Short version is that it doesn't matter how many physical slots are mounted on the motherboard or how many lanes can be allocated in the BIOS because the processor can simultaneously address a maximum of 40 PCIe 3.0 lanes worth of bandwidth under best conditions. There are zero performance gains once PCIe 3.0 devices saturate all these lanes (in fact, there are minor performance losses since the PLX processing part adds signal latency and complicates timings, the increased signal traffic adds packet envelope overheads, and the increased signal collisions/errors require more packets get resent). So you might be able to install four GPU cards in a PLX x16/x16/x16/x16 configuration but actual performance will be comparable to (actually worse than) a non-PLX x16/x8/x8/x8 configuration. (And, besides, even mighty 980Ti and TitanX GPU cards at full load can hardly saturate >8 PCIe 3.0 lanes in practice, especially if they're directly interlinked off the PCIe bus with a CF/SLI bridge.) Just gotta wait for future Intel processors with more integrated PCIe 3.0 lane controllers."

I realize that the PLX chip isn't a substitute for my CPU's lanes but the way many of these boards are configured, the PLX chip does allow me to use all the components of the board without automatically sacrificing PCIe slot speed, lanes themselves, RAM slots, and SATA connections, all of which are part of this give and take of PCIe. I mean look at part of the specs of the Designare board...

--------------------------------------------------------------
3 x PCI Express x16 slots, running at x16 (PCIEX16_1~3)
* The PCIEX8 slot shares bandwidth with the PCIEX16_1 slot. When the PCIEX8 slot is populated, the PCIEX16_1 slot operates at up to x8 mode.
* The U2_32G_2 connector shares bandwidth with the PCIEX16_2 slot. When the U2_32G_2 connector is populated, the PCIEX16_2 slot operates at up to x8 mode.
* The M2_32G connector shares bandwidth with the PCIEX16_3 slot. When the M2_32G connector is populated, the PCIEX16_3 slot operates at up to x8 mode.

1 x PCI Express x16 slot, running at x8 (PCIEX8)

1 x PCI Express x16 slot, running at x4 (PCIEX4)
* The SATA_EXPRESS connector shares bandwidth with the PCIEX4 slot. When the SATA_EXPRESS connector is populated, the PCIEX4 slot operates at up to x2 mode.
--------------------------------------------------------------

And that board actually has a PLX chip (I won't be getting that board). Two PLX chips however seem to enable the Asus workstation board (The first on the list) to not require sacrifice (It's also the most expensive of the lot) but there we're using two PLX PEX chips and so what's that going to mean for latency? This is my last question before I post my x99 extreme mobo breakdown thread.

Thanks for your answers.

 

[photonboy]

Hi,
PLX isn't needed with a CPU that can address enough PCIe lanes.

(with a CPU that only supports 16 lanes that would mean nothing left over for OTHER things besides graphics hence the requirement to have a PLX chip to link to the GPU lanes directly, but once you have 24 PCIe lanes supported that allows 2x8 for the GPU with 8 left over for other stuff.)

Provided the motherboard supports SLI, and can do 2x8 PCIe v3 for both GPU's there should be no issue.

So ignore PLX completely, and just focus on the motherboard you want. Even if it had PLX i don't know if it is automatically used if not needed. Even if it is, the real-world latency loss would not be noticeable.

I like the Sabertooth X99 in terms of Warranty. It is FIVE YEARS!