Question X570 Chipset - NVME sharing bandwidth with SATA/USB?

[CelticPharaoh]

Hello all,

I've been troubleshooting I/O stutters.

So to explain my storage configuration, I have 2x PCIe 4.0 NVMEs (1 directly connected to CPU lanes, 1 through the X570 chipset), 1x SATA HDD, 1x DVD Writer optical drive. Internal USB connections - Corsair AIO USB cable connected to USB 2.0 header.
External USB devices - keyboard, mouse, headset, 2x game controllers

So then I had read something about bandwidth sharing with CPU lanes (mostly Intel related) and I came across these diagrams:

https://imgur.com/a/b7tntpZ

View: https://imgur.com/a/b7tntpZ


But I'm struggling to determine if it could be that my chipset NVME is competing for CPU time with the other SATA drives and USB devices since there is a 4x CPU chipset link? So whenever there is a read operation on the chipset NVME, there is some deadlock happening between it and other devices. Or is it completely irrelevant?

Thanks

Edit;
Here are my specs;
CPU: AMD Ryzen 5950X | Cooler: Corsair H100i RGB PRO XT | GPU: EVGA RTX 3090 FTW3 ULTRA | RAM: TEAMGROUP T-Force Xtreem ARGB 3600MHz CL14 32GB (2x16GB) PC4-28800 Dual Channel DDR4 DRAM Desktop Gaming Memory Ram (White) - TF13D432G3600HC14CDC01 | SSD: 2x 2TB Samsung 980 PRO M.2 NVMe SSD | HDD: Seagate BarraCuda Pro 10TB | Motherboard: ASUS ROG Crosshair VIII Dark Hero X570 (BIOS 4201) | PSU: be quiet! Dark Power 12 1000w | Case: Fractal Design Define 7 XL | OS: Windows 10

 

[Lutfij]

I've been troubleshooting I/O stutters.
When posting a thread of troubleshooting nature, it's best that you include the specs to your build in the thread's body. I noticed that you've got some parts listed in your sig space, the problem with that is due to the fact that sig space specs can and will change over time. When that does happen the thread and any relevant solutions or suggestions are rendered moot to the person in the same boat as you're in now. I've edited in your specs into your thread's body.

That being said, this is what caught my eye;
Motherboard: ASUS ROG Crosshair VIII Dark Hero X570 (BIOS 4201)

ROG Crosshair VIII Dark Hero | Motherboards | ROG United States

AMD Ryzen 3000 series ATX motherboard with Aura Sync, SupremeFX, ROG Audio, Dual M.2, Realtek LAN, Wifi , M.2 heatsink and USB 3.2 Gen 2

rog.asus.com

cross referencing the support site for the motherboard(assuming you're trying to troubleshoot that motherboard in the build) then there is no BIOS version 4201 listed on support site, apart from 4006 and 4402. If you're indeed on the BIOS version listed, in your specs, you likely have a BIOS version that was deemed unstable or faulty.

 

[CelticPharaoh]

I've been troubleshooting I/O stutters.
When posting a thread of troubleshooting nature, it's best that you include the specs to your build in the thread's body. I noticed that you've got some parts listed in your sig space, the problem with that is due to the fact that sig space specs can and will change over time. When that does happen the thread and any relevant solutions or suggestions are rendered moot to the person in the same boat as you're in now. I've edited in your specs into your thread's body.

That being said, this is what caught my eye;
Motherboard: ASUS ROG Crosshair VIII Dark Hero X570 (BIOS 4201)

ROG Crosshair VIII Dark Hero | Motherboards | ROG United States

AMD Ryzen 3000 series ATX motherboard with Aura Sync, SupremeFX, ROG Audio, Dual M.2, Realtek LAN, Wifi , M.2 heatsink and USB 3.2 Gen 2

rog.asus.com

cross referencing the support site for the motherboard(assuming you're trying to troubleshoot that motherboard in the build) then there is no BIOS version 4201 listed on support site, apart from 4006 and 4402. If you're indeed on the BIOS version listed, in your specs, you likely have a BIOS version that was deemed unstable or faulty.

Hi Lutfij,

Thank you for letting me know. My system is exactly as noted in my signature.

Good catch about the BIOS! I would have never thought about this since up until recently, 4201 was relatively recent and still listed on Asus's website but the fact that it was taken down is a bad sign. I'll try going up to 4406 and then go see what happens

 

[Lutfij]

You're also on Samsung drives, make sure that they are on the latest firmware using Samsung's magician app.

 

[CelticPharaoh]

You're also on Samsung drives, make sure that they are on the latest firmware using Samsung's magician app.

Yeah, I already updated to the latest firmware a month or two ago.

So I managed to upgrade the BIOS. Strangely, however, it was marked as version "4402" in BIOS even though the download was noted as "4406" but it updated successfully.

The behaviour is unchanged after the update. Still getting microstuttering :/

Just to note, I have already RMA'ed the PSU, CPU and tried a different RAM kit, GPU, none of which resolved the issue. I'm dreading to think that it could be the motherboard because of the hassle removing it from the system but that's the only part I haven't swapped out at this point

 

[Lutfij]

The listed latest BIOS on support site is 4402, I also clicked on the download to see that it's instructed to be saved as 4402. I opened the zipped file and it's also showing up as 4402. I don't know why you'd be seeing 4406 in any of those places.

Did you clear the CMOS after verifying that the BIOS was successfully flashed to the latest version?

 

[hotaru.hino]

I'm not sure if LatencyMon is the right tool to troubleshoot this, but it's the only thing on my head since it measures system latency.

In any case, I don't think the other peripherals would be causing an issue because:

• PCIe uses a packetized data transfers, meaning all of the data sent back and forth between PCIe nodes are in fixed block sizes. It's very similar to how network traffic works. I haven't read into the spec much but I'm also certain there's no real priority involved. That is, the PCIe "network" treats all packets equally so nobody gets to cut in line.
• Unless you're transferring something over one of the SATA ports constantly, none of the other devices will generate enough data to cause human perceptible latency against what the NVMe drive is doing, in theory anyway.

Also I'm struggling to figure out what BIOS's role in all of this is, since BIOS contains no software that influences the chipset outside of configuring which ports are in use or not. It wouldn't have any role in how the chipset behaves or performs outside of that.

 

[CelticPharaoh]

The listed latest BIOS on support site is 4402, I also clicked on the download to see that it's instructed to be saved as 4402. I opened the zipped file and it's also showing up as 4402. I don't know why you'd be seeing 4406 in any of those places.

Did you clear the CMOS after verifying that the BIOS was successfully flashed to the latest version?

Sorry, I think I misread the BIOS versions on the Asus page. It is 4402. I think it was because I saw 4006 so I combined the two!

The BIOS was cleared/reset after the update so I don't think its a BIOS setting. I've already tried running RAM at both JEDEC (2400Mhz) and D.O.C.P. (3600Mhz) as well as a different RAM kit sometime ago.

I'm just trying to understand if its an issue with my hardware, is it a software problem, or even a design issue with one of the components. If it is a hardware fault, keeping in mind that I've RMA'ed PSU, CPU, tried a different GPU, different RAM, then I'll see about an RMA. The only components I haven't changed are the motherboard, NVME drives and case. I doubt its a faulty NVME drive, given that the behaviour is the same, regardless of which SSD the game is running on, CPU NVME or chipset NVME.

For the motherboard and case, I would really like to make certain that there is an actual issue with either of them before I dismantle my system again. I don't like the idea of removing and installing the CPU and GPU each time, wearing out PCIe pins or risk bending CPU pins or getting thermal paste on them

 

[CelticPharaoh]

I'm not sure if LatencyMon is the right tool to troubleshoot this, but it's the only thing on my head since it measures system latency.

In any case, I don't think the other peripherals would be causing an issue because:

• PCIe uses a packetized data transfers, meaning all of the data sent back and forth between PCIe nodes are in fixed block sizes. It's very similar to how network traffic works. I haven't read into the spec much but I'm also certain there's no real priority involved. That is, the PCIe "network" treats all packets equally so nobody gets to cut in line.
• Unless you're transferring something over one of the SATA ports constantly, none of the other devices will generate enough data to cause human perceptible latency against what the NVMe drive is doing, in theory anyway.

Also I'm struggling to figure out what BIOS's role in all of this is, since BIOS contains no software that influences the chipset outside of configuring which ports are in use or not. It wouldn't have any role in how the chipset behaves or performs outside of that.

I've heard and used LatencyMon before so it might be worth giving this another go.

Thanks for the clarification on how PCIe data transfers occur. I suppose what I'm thinking is that the NVME, SATA HDD and USB devices are competing for those 4 PCIe lanes with the CPU

What I'm thinking is that because NVME drives generally have very fast I/O operations, in the case of the 980 pro specifically up to 7GB per second, its saturating those 4 PCIe lanes, causing stutters when any SATA/USB devices try to send data, it interrupts the NVME. But say if I was running SATA SSDs or SATA HDDs then it might be a different story, although I did try disabling SATA ports in the BIOS and it didn't resolve the stuttering so I'm kinda lost at this point as to what the issue could be

 

[hotaru.hino]

So the only other thing I can think of to try before dismantling the computer again is if fTPM is enabled. If it is, then there's a fTPM stuttering bug that got fixed in AGESA version 1.2.0.7. The only non beta version of BIOS that has this or later is version 4402.

What I'm thinking is that because NVME drives generally have very fast I/O operations, in the case of the 980 pro specifically up to 7GB per second, its saturating those 4 PCIe lanes, causing stutters when any SATA/USB devices try to send data, it interrupts the NVME. But say if I was running SATA SSDs or SATA HDDs then it might be a different story, although I did try disabling SATA ports in the BIOS and it didn't resolve the stuttering so I'm kinda lost at this point as to what the issue could be

The data payload in PCIe packets are limited to 4KiB (which makes sense, as this is typically the minimum addressable block of data). So even if the NVME drive is saturating the PCIe link, if say your keyboard or mouse wants to send something, it'll only cause an interrupt of at most 510 nanoseconds. Though I can't imagine a keyboard or mouse that needs to send any more than 4KiB, let alone getting close to that.

Also stuttering would imply the game is waiting for something essential to be done. I would argue once you're actually playing the game, storage and input devices aren't necessarily essential. If it was storage, it would generally happen if you're going into new areas because you hit a loading zone. There's likely something tying up the CPU for some reason.

 

[CelticPharaoh]

So the only other thing I can think of to try before dismantling the computer again is if fTPM is enabled. If it is, then there's a fTPM stuttering bug that got fixed in AGESA version 1.2.0.7. The only non beta version of BIOS that has this or later is version 4402.


The data payload in PCIe packets are limited to 4KiB (which makes sense, as this is typically the minimum addressable block of data). So even if the NVME drive is saturating the PCIe link, if say your keyboard or mouse wants to send something, it'll only cause an interrupt of at most 510 nanoseconds. Though I can't imagine a keyboard or mouse that needs to send any more than 4KiB, let alone getting close to that.

Also stuttering would imply the game is waiting for something essential to be done. I would argue once you're actually playing the game, storage and input devices aren't necessarily essential. If it was storage, it would generally happen if you're going into new areas because you hit a loading zone. There's likely something tying up the CPU for some reason.

I disabled Secure Boot/fTPM a long time ago and don't use BitLocker or the like. It does seem to be tied to loading in-game but CPU has already been RMA'ed and behaviour is the same.

I guess I could try running a basic boot/diagnostic startup. I'm able to reproduce the issue easily so I'll know pretty quickly if it is some other process hogging the CPU or not, causing the issue.

One last thing, could it possibly be a power delivery issue with the PSU?

While I know be quiet PSUs are meant to be very good, the 1000w unit I have is analogue and not digital, which the Dark Power Pro 12 1200w and 1500w are, and maybe 1000w isn't quite enough wattage? Thanks to NVIDIA's stupid 2x/3x transient power spikes

I have also heard a 'click' noise whenever the GPU usage is high (350w+) but it only clicks once and I don't hear it again. I've also heard the click with my original PSU before I RMA'ed it so its not a 'bad' PSU per se unless I'm very unlucky and got 2 bad PSUs

 

[hotaru.hino]

I guess I could try running a basic boot/diagnostic startup. I'm able to reproduce the issue easily so I'll know pretty quickly if it is some other process hogging the CPU or not, causing the issue.

If you have time to learn how to use it, I suggest looking into Windows' Performance Monitor. It's like Task Manager's Performance tab on steroids. You can even have it track stats per application rather than system wide.

One last thing, could it possibly be a power delivery issue with the PSU?

While I know be quiet PSUs are meant to be very good, the 1000w unit I have is analogue and not digital, which the Dark Power Pro 12 1200w and 1500w are, and maybe 1000w isn't quite enough wattage? Thanks to NVIDIA's stupid 2x/3x transient power spikes

I have also heard a 'click' noise whenever the GPU usage is high (350w+) but it only clicks once and I don't hear it again. I've also heard the click with my original PSU before I RMA'ed it so its not a 'bad' PSU per se unless I'm very unlucky and got 2 bad PSUs

Poking around on the internet tells me the 3090 can get as bad as 600W, however the rest of the system shouldn't add more than 200-250W, so I don't think the spikes would cause a problem per se. Besides, I imagine if the spikes were causing problems, it'd effect the entire system in odd ways rather than just hiccuping.

EDIT: Further poking around on the internet to look in another issue led me to find someone who had a 6900 XT with a 850W PSU having latency issues. They worked around it by setting a frame rate cap.

See if that helps.