[SOLVED] M.2 Raid 0 scaling issues

[purple stain]

Hello Everyone!

I just installed a second M.2 that I have setup for Raid 0 however when I'm running a speed test it's only reflecting the speed of a single drive. Not sure whats going on with it. Any ideas? I thought it might have been the PCI-E Lanes not being allocated properly but everything checks out. Also installed latest Intel RST drivers.

https://imgur.com/a/wisa9py

View: https://imgur.com/a/wisa9py


Asrock Z370 Gaming K6
Samsung 970 Evo M.2 500GB
Samsung 970 Evo M.2 500GB
GTX1080 Ti

 

[popatim]

Looks like you're limited by the bus between the chipset and the CPU which is also an x4 pcie connection.

 

[purple stain]

How is that possible? I thought this was possible on the Z370 Chipset.

This is the page from the motherboard manual regarding the M.2 operation

https://imgur.com/a/TyUWnK9

View: https://imgur.com/a/TyUWnK9

In the first paragraph it states "support SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen3 x4 (32 Gb/s)"

I'm going to be naive here and believe they mean it's 32 Gb/s per M.2 Slot which it should be, sharing PCI-E lanes and all. I thought the Z370 manged the SATA Controllers and M.2 was PCI-E. But Apparently they are doing it backwards here.

 

[popatim]

See the overview image here
https://www.anandtech.com/show/11860/z370-motherboards-asus-asrock-ecs-evga-biostar-msi-gigabyte

See that DMI 3.0 bus between the CPU and the chipset - it's basically an x4 pciex bus; there's your bottleneck.

View: https://www.youtube.com/watch?v=WqpX-kiM8iQ
Jump to about 22:00 to watch them install W10 to a raid0 and benchmark it.

 

[USAFRet]

Additionally...
https://www.tomshardware.com/reviews/samsung-950-pro-256gb-raid-report,4449.html

 

[purple stain]

Wow. Thanks everyone. I had no idea the DMI handled the communication the the Southbridge. I thought it all pick backed off of PCI-E. And that's the answer I've been looking for. Thank you. Now I guess it does make sens now why they would include two M.2 ports not to do what I'm doing but to use M.2 SSD's.

So if I got a M.2 PCI-E card that would be a work around for the DMI bridge?

Also does Ryzen suffer the same issue with the DMI? Their boards have PCI-E 4.0.

 

[Maxxify]

Welcome to Intel consumer boards, where all M.2 sockets go over the chipset! (which adds some latency too, by the way)

You can use CPU lanes by diverting from the primary GPU (the GPU will run x8), most likely the third full-length PCIe slot goes over the chipset so you'd have the same bottleneck.

Older AMD boards have x4 PCIe 3.0 CPU lanes dedicated to a primary M.2 socket, any other M.2 socket will be over chipset and limited to at most x4 PCIe 2.0 speeds or equivalent (x2 PCIe 3.0) with minor exceptions that pull from GPU (it'll run x8 on those boards). Newer AMD boards, X570-based for example, have far superior options. It's in fact possible to run three x4 PCIe 3.0 M.2 drives at full speed simultaneously, and if you do take from the GPU you can run up to 5 in that manner while leaving x8 PCIe 4.0 for a GPU (upcoming generation).

Example from my X570 Aorus Master: I have 1TB EX920 in the dedicated M.2 socket (CPU lanes) for OS/apps, 2x1TB SN750s in RAID-0 (Hyper), 1x1TB SN550, and 1x2TB EX950, and all are are full speed with in fact bandwidth to spare for 3x500GB SATA SSD RAID-0 over chipset.

 

[purple stain]

Awesome! Thank you for all that valuable information. So if you were to do it on the AMD side, does all the pcie lanes come from the cpu or is the x570 chipset just able to handle that much. Actually just looking at the specs a Ryzen 7 3700x has 24 pcie lanes, gen 4 to boot. I can see the money separation now from Intel. Making the max consumer lanes you can have is 16 unless you go with the HEDT platform. Damn Intel!

Well I've been looking at the Ryzens for a while now. Does anyone know if the current x570 chipset will support a Ryzen 4000 series desktop cpu? That would definitely make me decision easier.

Thanks again Maxxify

 

[popatim]

Let us know the progress and results of your new build when you get to it.

 

[Maxxify]

Awesome! Thank you for all that valuable information. So if you were to do it on the AMD side, does all the pcie lanes come from the cpu or is the x570 chipset just able to handle that much. Actually just looking at the specs a Ryzen 7 3700x has 24 pcie lanes, gen 4 to boot. I can see the money separation now from Intel. Making the max consumer lanes you can have is 16 unless you go with the HEDC platform. Damn Intel!

Well I've been looking at the Ryzens for a while now. Does anyone know if the current x570 chipset will support a Ryzen 4000 series desktop cpu? That would definitely make me decision easier.

X570 will support the upcoming Zen 3 chips and in fact so will older boards. There's also options available from B550 that might meet your needs. Please refer to this thread if you want to narrow down a board choice.

x4 lanes come from the CPU to a primary M.2 socket, these are dedicated. x16 go to the GPU in one or more sockets - you can siphon away 8 as you do on Intel boards, with most GPUs being fine with just x8 PCIe lanes. Note that you can bifurcate (halve) this to 4x/4x for two NVMe SSDs depending on the hardware. AMD's newer boards are better here since x8 PCIe 4.0 is twice the bandwidth and upcoming cards will be PCIe 4.0 (some already are). The last x4 lanes go to the chipset. In the case of older AMD boards, this is switched to x8 PCIe 2.0 which is why x4 PCie 2.0 bandwidth is the maximum for M.2 (lanes are lanes, although PCIe 2.0 vs. 3.0 has different bandwidth and encoding, but you can switch x4 PCIe 2.0 to x2 PCIe 3.0), however on newer boards with newer CPUs this is x4 PCIe 4.0 which is equivalent to x8 PCIe 3.0 (thus, two PCIe 3.0 NVMe SSDs). Intel works the same, they just don't have the x4 dedicated to a M.2 socket and have x4 PCIe 3.0 upstream instead on chipset. This doesn't cover multiplexing which means virtual PCie lanes (downstream) as you're limited by upstream, however going over the chipset adds some latency.

If your goal is to maximize sequential bandwidth with SSDs, X570 is currently the best platform. X670 will likely be better as the X570 chipset is kind of wonky for storage performance in my opinion, but still quite ample if you understand how to make use of it. In fact, it's possible to run even more than I do on certain boards if you have a weaker discrete GPU. For example the ASUS Pro WS X570-ACE can have a x8 PCIe 3.0 GPU in a chipset PCIe slot and run up to 5 PCIe 4.0 NVMe SSDs (4x via hyper, 1 in a dedicated socket).

 

[USAFRet]

I'm really interested in seeing the user facing results in different workloads.

 

[Maxxify]

I'm really interested in seeing the user facing results in different workloads.

Here is an article I posted recently that covers the topic to some extent but I agree it sees little benefit in general use. The primary reason I RAID SN750s is for sustained writes (double the SLC cache, then more importantly high TLC steady state speeds) and SATA SSDs purely for sequentials (to transfer to/from my NVMe SSDs). In general I feel those that go for sequentials as you would find in Gen4 drives for example are people who should really already be HEDT at the minimum. IOPS and high QD, even more so.

 

[USAFRet]

Here is an article I posted recently that covers the topic to some extent but I agree it sees little benefit in general use. The primary reason I RAID SN750s is for sustained writes (double the SLC cache, then more importantly high TLC steady state speeds) and SATA SSDs purely for sequentials (to transfer to/from my NVMe SSDs). In general I feel those that go for sequentials as you would find in Gen4 drives for example are people who should really already be HEDT at the minimum. IOPS and high QD, even more so.

Interesting. Thanks.

 

[purple stain]

Thanks everyone again for all the help! Learned a lot through this, as always .

While I've definitely found what my problem is I now have some decisions to make.

In the end I'm really torn because it's one of those moments in the cycle from Intel that they'll be going to a new Architecture and that's always fun. Getting a 1200 series board from Intel now would just make it a drop in CPU replacement for PCI-E 4.0 in the future. But that future is probably very distant. The Ryzen option lands me comparable performance with Intel's current offerings with the ability for PCI-E 4.0. And gives me the ability to drop in a 4000 series CPU which I honestly think is going to finally start putting AMD out in front of Intel when it comes to gaming. All my opinions though.

One a side note, I have noticed since getting a gigabit connection that when downloading games my processor is maxed out when trying to uncompress them. I imagine additional cores would help with this as well.

Thanks again all!