[SOLVED] How does PCIE slots x16/x8/x4/x1 and storage work exactly? Need help with explaining the traffic...

[zze86]

I have a AsRock X570 Taichi motherboard with a AMD 3900XT. There are:

3 x PCIE 4.0 x 16 slots (PCIE1/PCIE3PCI5)
[single at x 16 (PCIE1)
dual at x 8 (PCIE1) / x8(PCIE3)
triple at x8(PCIE1) / x8(PCIE3) / x4 (PCIE5)]
2 x PCIE 4.0 x 1 slots (PCIE2, PCIE4)

and for storage:

1 x Hyper M.2 socket (M2_1) supports SATA and M.2 PCI Express module up to Gen4x4
1 x Hyper M.2 socket (M2_2) supports SATA and M.2 PCI Express module up to Gen4x4
1 x Hyper M.2 socket (M2_3) supports SATA and M.2 PCI Express module up to Gen4x4
*If M2_3 is occupied, PCIE5 slot will be disabled


I get the * part of it. If M2_3 is used then PCIE5 no longer works. Basically, M2_3 takes up all the traffic lanes. If I am understanding this correctly, the M.2 sockets operate off the PCIE lanes thus if I have all three slots occupied, it's possible that storage devices could be taking up to 12 PCIE traffic lanes...?

If I occupy ONLY PCIE1 then that device has access to all 16 PCIE lanes....which could potentially be throttled down to just x4 lanes if all three M.2 sockets that up their max x12 lanes...?
If I occupy two PCIE slots (PCIE1 and PCIE3) then both devices get only x8 lanes...which could potentially be throttled down by the M.2 sockets...?

Here's where I'm losing it:
If I take out M.2_3 - giving me access to PCIE5 - and I occupy PCIE1, PCIE3 and PCI5 then both devices on PCIE1 and PCI3 get x8 lanes each but PCIE5 only gets x4 lanes. <- That's more than x16 lanes...? Or are they only allowed these many lanes MAXIMUM?
Let's say I somehow was able to physically occupy all the PCIE slots (PCIE1, PCIE2, PCIE3, PCIE4 and PCIE5). PCIE1 and PCI3 get x8 lanes each but PCIE5 only gets x4 lanes. PCI2 and PCI4 each get 1 lane each. <- again more than 16 lanes. Or is this the MAX number of lanes they are allowed? So if PCIE1 has heavy traffic and there are no other loads, PCIE1 gets a MAX of x8 lanes...?


EDIT:
Just found a X570 block diagram but I'm still confused...there are x20 PCIE lanes?

 

[Colif]

CPU can handle some of the lanes - With Ryzen, the CPU generally has 1x16+x4 with the x4 used for drives, in addition to the 16 from the chipset.

4x nvme is likely talking about fact NVME use 4 lanes

that chart seems familiar - https://www.tomshardware.com/news/a...tal of 24 CPU,that can be used concurrently."

 

[drea.drechsler]

....

EDIT:
Just found a X570 block diagram but I'm still confused...there are x20 PCIE lanes?

20 from the chipset, plus 24 from the CPU of which 4 are used for chipset to CPU traffic leaving 40 available. Usually, 16 from the CPU are dedicated to the GPU (PCIe1) and 4 from the CPU dedicated to an NVME (M.2_3). The rest can be split out according to how the motherboard mfr. allows in it's design.

That Taichi lane allocation chart is confusing but they probably have to split up lanes from the CPU separately from lanes from the chipset and can't combine them either. So it appears the CPU lanes can only be allocated to PCIe1, 3 and 5 and M.2_3. With that in mind it starts to make some sense.

So:
3 x PCIE 4.0 x 16 slots (PCIE1/PCIE3PCI5)
[single at x 16 (PCIE1) means one single x16 device at PCIE1
dual at x 8 (PCIE1) / x8(PCIE3) means two x8 devices, allowing crossfire or SLI
triple at x8(PCIE1) / x8(PCIE3) / x4 (PCIE5)] means 3 devices, 2-x8 and 1-x4, but only so long as M.2_3 is not being used because:

*If M2_3 is occupied, PCIE5 slot will be disabled means this is the lowest latency, direct to CPU data path for NVME data/code

All the above lanes are sourced from the CPU. The other allocation rules for PCIe2 and 4 and M2_1 and M2_2 relate to lanes that source from the chipset and so are different.

Also, something often overlooked is that while you might have a PCIe x16 device on the chipset, data traffic that needs to get to the CPU is still limited to x4 at best since that's the chipset to CPU data path bandwidth. That's why it's "better" to put your GPU on a CPU-sourced PCIe slot even if another x16/16 slot is available.