So does anyone understand the PCIe bus enough to make sense of WD's explanation? Sounds incomplete. Who is setting the 128 byte MPS? Is the packet size negotiated and the SN850 wasn't doing that? Just curious.
MPS, Maximum Payload Size, dictates the size of the individual data packets on PCIe (TLP; payload). Simply put, a larger payload size increases efficiency of the PCIe link because you can transmit more data per "chunk", and reduce link overhead (requests, acknowledgement, etc).
Here is an excellent overview of MPS, and the relation with link efficiency (pdf warning!):
Thst said, the difference between MPS=256B vs. 128B is about 200-300MB/s, when the link is at 100% utilization. It makes no difference to throughput, from link point of view, until you approach link saturation.
WD fixed it fast! Nice work!
(Oops, didn't mean to reply to post!)