Question PCIe M.2 Adapter Board in an X99 System ?

[dcairns]

So I want to add some more fast M.2 drives on my X99 system and the PCIE Adapter boards seem the obvious route. My board supports one M.2 drive, but I would like to be able to add more. So I have poked around, reading/watching stuff on how this works, and looks like it would be fine on my system, so I am looking for a check on my research here.

So my idea is a something like the ASUS Hyper M.2 X16 PCIe 4.0. (open to other/cheaper suggestions) Yes, I know my system only does PCIE 3.0, so I might be over spending on that, but being able to add more M.2 NVME drives as I can afford them, is an interesting idea. The idea would be to have the drives in the ASUS Hyper board appear as one drive or separate drives, no RAID.

My understanding of PCIE lane usage:

i7-6900K supports 40 PCIE lanes

Asus X-99 A II Motherboard supports PCIE 3.0 and 40 PCIE lanes.

The manual rather vaguely breaks down the PCIE slot/lane support like this:

GPU Cards PCIe Slot 1 PCIe Slot 3 PCIe Slot 4

1 x16

2 N/A x16 x16

3 x8 x16 x8

So by my understanding, the M.2 slot takes x4, the 1070 Ti takes 16 making at total of 20 lanes in use, with 20 more free. A ASUS Hyper M.2 X16 PCIE 4.0 with 4 SSD drives in it would take an additional 16 leaving me with 4 remaining.

Just to show how much my work drive could use some help,

Crystal Mark rates my M.2 boot drive as: Seq Q32T1 1810/1254

My SATA SSD work drive as: Seq Q32T1 528/507

My SATA mechanical HDD as: Seq Q32T1 217/211


System specs::

Windows 10 22H2 19045.4651 (never going to 11, planning on jumping ship to Linux)

i7-6900k @4Ghz

Asus X-99 A II Motherboard

Water cooled CPU, 96GB RAM

NVIDIA GeForce GTX 1070 Ti (8Gb) feeding three 27" monitors

Storage:

One built in M.2 NVME slot in use as the boot drive

2TB SSD Drive (SATA)

1TB SSD Drive(SATA)

Three spinning 8 TB Drives (SATA)

12TB spinning 12 TB Drive (SATA)

Offline 8TB backup drive (USB 3.1) for essentials.

The system is used for photo and video editing workstation general PC use and some gaming. Yes, I am a digital pack rat.

 

[dcairns]

Well, I went ahead and got the ASUS Hyper M.2 X16 PCIe 3.0 X4 Expansion Card V2 and a SAMSUNG 990 PRO SSD 4TB PCIe 4.0 M.2. So I will just find out on my own. I am pretty sure it will work just fine. Perhaps I overspent on the M.2 drive, but 4.0 will give it some future life.

 

[Maxxify]

You need single-slot PCIe bifurcation to make use of the Hyper. It's not the solution you want for that machine. The manual actually isn't vague, but to elaborate, the 6900K is a 40-lane CPU so for the 3.0 PCIe slots can do x16, x16/x16, or x8/x16/x8. You can use one of these for a discrete GPU and another for an M.2 expansion card that has a PCIe switch/controller on it. Choice of AIC depends on what kind of bandwidth you want. If you're going for the maximum x16 3.0 with four drives, then it's something like the SSD7104/SSD7105. If x8 3.0 is enough or you want to use all three PCIe slots with the x16 for the GPU, then the SSD7204. (just examples, there's probably better options)

 

[dcairns]

Well, this is the kind of info I needed before getting impatient and pressing the "Buy" button . I was able to cancel the order for the Asus add in card. After reading your post and some more "learning", I realized the Asus card was all about RAID 0 (striping) and there is no mention of bifurcation in my motherboard documentation. I can appreciate the performance of doing that, but having tried RAID ) 20 years ago, I am not fond of it. Too temperamental, but maybe it is better now. Anyways, I am just looking to get the best performance I can out of the M.2 SSD. So back to looking at add in card to carry the M.2. Maybe just the single M.2 would be ok.

 

[Maxxify]

Well, this is the kind of info I needed before getting impatient and pressing the "Buy" button . I was able to cancel the order for the Asus add in card. After reading your post and some more "learning", I realized the Asus card was all about RAID 0 (striping) and there is no mention of bifurcation in my motherboard documentation. I can appreciate the performance of doing that, but having tried RAID ) 20 years ago, I am not fond of it. Too temperamental, but maybe it is better now. Anyways, I am just looking to get the best performance I can out of the M.2 SSD. So back to looking at add in card to carry the M.2. Maybe just the single M.2 would be ok.

Technically, your board does bifurcation across multiple PCIe slots, but for that type of card you need it do it with/within a single slot. Many consumer boards can do this but with HEDT like yours you have a lot more CPU lanes to work with so you can use something higher-end. You just need an AIC with hardware on the PCB, most commonly a PCIe or packet switch. These AICs cost more.

RAID for either type of AIC is not necessary at all. In most cases, it would be software RAID, anyhow, not hardware RAID. So it's not correct to say the Hyper or my suggested alternative would force you into RAID of any kind. If your goal is to add up to four drives with as much bandwidth as possible, and bandwidth is only one aspect of performance mind you, then you would need to take my suggestion.

 

[thestryker]

You basically have two options here:

1. Straight PCIe to M.2 adapter which is a 1:1 so you can add a single M.2 drive into a single PCIe slot
2. An adapter with a PCIe switch on it which will handle the bifurcation on the card itself. The cost would depend on how many drives you wanted supported and what amount of bandwidth you wanted allocated

example of the first: https://sabrent.com/products/ec-pcie
example of the second: https://www.newegg.com/riitop-dul-nvtpce8x-pci-express-controller-card/p/17Z-0061-00097

 

[dcairns]

Well, those HighPoint 7104 or 7105 are a bit pricey for now. So I just punted and got a single SABRENT NVMe M.2 SSD to PCIe X16/X8/X4 Card for $18. I can defer the idea of a multi SSD card for later when I can afford more M.2 SSDs. This will out perform the existing 2TB SATA SSD that I use at the moment. And maybe those nicer multi M.2 cards will get cheaper by then.
Thank you for the help

 

[Bob.B]

Well, those HighPoint 7104 or 7105 are a bit pricey for now. So I just punted and got a single SABRENT NVMe M.2 SSD to PCIe X16/X8/X4 Card for $18. I can defer the idea of a multi SSD card for later when I can afford more M.2 SSDs. This will out perform the existing 2TB SATA SSD that I use at the moment. And maybe those nicer multi M.2 cards will get cheaper by then.
Thank you for the help

Depending on what you do with the pc don't be surprised if you don't see much a diff in perf after you do this swap.

 

[dcairns]

Since I overspent on the 4TB M.2 SSD, for the moment I am going to try the single M.2 SSD on the single adapter card. The card was cheap enough ($18). However given the cost of adding more three more matching 4TB M.2 SSD's, I may just do a two M.2 SSD card. That would keep the add in card cost down and only need to buy another 4TB M.2 SSD for a 2x speed increase.
I am still struggling with the proper way to vet the add in cards for the bifurcation.
My use case for this "drive" will be photo editing and video editing, so large sequential read/write activity.

 

[USAFRet]

and only need to buy another 4TB M.2 SSD for a 2x speed increase.

You're thinking of a RAID 0 with these?

Apart from artificial benchmarks, RAID 0 does not benefit like it did with HDDs.

In some instances, it is even slower.

 

[dcairns]

Perhaps I misunderstood the recommendation for the SSD7104/SSD7105 add in cards. I thought their performance boost was due to the RAID 0 striping.

 

[thestryker]

I am still struggling with the proper way to vet the add in cards for the bifurcation.

PCIe 3.0 cards will all be using ASM2812 or ASM2824 controllers for the most part. These should be available starting at around $100 USD, but potentially less if you're willing to order direct from China via AliExpress or the like. I linked an example in my prior post.

PCIe 4.0+ ones you're probably talking a significant investment that doesn't make sense. PCIe switches are very expensive so I'd expect $300+ for pretty much anything in this category.

 

[USAFRet]

Perhaps I misunderstood the recommendation for the SSD7104/SSD7105 add in cards. I thought their performance boost was due to the RAID 0 striping.

Traditionally, yes. RAID 0 + HDD was a BIG performance boost, in the right circumstances.

But with the advent of SDDs, not so much.
Especially in typical consumer use.

SSDs main benefit over HDD is the near 0 access time. That is across all SSDs.
Small file access, which is what we do the vast majority of the time, benefits not at all from that RAID 0.

Often, even just between the different PCIe variants.

In a blind test, it is often hard to tell the user facing difference between PCIe 3.0, 4.0, 5.0 drives.

Now...if you were routinely transferring large blocks of data between 2 such RAID 0 arrays, you'd see a difference. Potentially a big diff.
But that is not what we do.

If RAID 0 + SSD were such a big performance boost, we'd all be doing it.

Here is a couple of older tests, showing basically NO difference.
I've not seen anything to change my mind.
http://www.tomshardware.com/reviews/samsung-950-pro-256gb-raid-report,4449.html
http://www.tomshardware.com/reviews/ssd-raid-benchmark,3485.html

 

[dcairns]

Yes, I have been looking at that RIITOP Dual NVMe PCIe Adapter you linked earlier. That is a lot more affordable than the SSD7104/SSD7105 option. And perhaps fits my situation better at this point.

Looking over those RAID links I can see the mixed results of RAID 0 and it got me to do a test on my existing system. This highlights the issue I am trying to resolve. A 664KB Photoshop file takes 29 seconds to save to my existing SATA SSD
Strangely it takes the same time to save to my on-board M.2 drive and SATA Spinning drive. So clearly this process is not bound by disk throughput. CPU utilization only has a slight up tick, so not CPU bound either. A file copy using Windows Explorer, from M.2 drive to SATA SSD, is finished in less than1 sec (better than my reflexes and stop watch can manage). Still close to 1 sec if I use a 2.5GB file. Looks like the blame rests squarely with Adobe on that one. I guess I should not be surprised. It just leaves me wondering what is taking so long with a photoshop file save.

But back to the topic, am I missing something on how those SSD7104/SSD7105 cards could boost performance?

 

[thestryker]

But back to the topic, am I missing something on how those SSD7104/SSD7105 cards could boost performance?

The only way they'd boost performance over individual drives is if you were using some form of RAID.

I do think those may be the only type of card with 4x M.2 ports and bifurcation on the card itself. I don't know that there's a 4 port version of that Riitop card (or similar), but at the same time you have the slots available and buying two of the Riitop cards is cheaper than a single 7104 so if you weren't going to leverage RAID it would make sense to just go that route.

 

[USAFRet]

Yes, I have been looking at that RIITOP Dual NVMe PCIe Adapter you linked earlier. That is a lot more affordable than the SSD7104/SSD7105 option. And perhaps fits my situation better at this point.

Looking over those RAID links I can see the mixed results of RAID 0 and it got me to do a test on my existing system. This highlights the issue I am trying to resolve. A 664KB Photoshop file takes 29 seconds to save to my existing SATA SSD
Strangely it takes the same time to save to my on-board M.2 drive and SATA Spinning drive. So clearly this process is not bound by disk throughput. CPU utilization only has a slight up tick, so not CPU bound either. A file copy using Windows Explorer, from M.2 drive to SATA SSD, is finished in less than1 sec (better than my reflexes and stop watch can manage). Still close to 1 sec if I use a 2.5GB file. Looks like the blame rests squarely with Adobe on that one. I guess I should not be surprised. It just leaves me wondering what is taking so long with a photoshop file save.

But back to the topic, am I missing something on how those SSD7104/SSD7105 cards could boost performance?

While not Adobe, this is the timing of writing the same 10 minute video file out to 3 difference SSD types.
Fast PCIe 40. Slow PCIe 3.0, and SATA III SSD.

The times are virtually identical. The rest of the system has far more impact than the target drive.

 

[USAFRet]

The only way they'd boost performance over individual drives is if you were using some form of RAID.

Even then, it doesn't.

Only if you were copying big blocks of sequential data, to another like array.

Apart from big benchmark numbers, RAID 0 + SSD is mostly useless.

 

[dcairns]

I was doing some additional searching and slow photoshop file writing seems to be a common complaint in recent years. I already had most of the recommended settings to avoid the issue (lots of RAM allocated for PS, large cache level, etc) and compared to others complaining of their write times being measured in minutes, not seconds, I guess I am not doing too bad. I guess another reason to work on my planned migration away from Windows and Adobe.

I am surprised that all these advancements in disk interface speed have not done much for daily use performance. I did expect the usual thing, where benchmarks are optimistic and real life is not as good, but I would have thought you could gain some significant fraction of performance gain.
I think I will return the unopened 4TB M.2 SSD Drive and reconsider my plans here. If my existing 2TB SATA SSD performs as well as the new 4TB M.2 SSD drive, the additional space on the new 4TB SSD is not that big of a deal to me. I routinely move old projects off my working drive to spinning disks for archive.
A better use for the M.2 interface card might be for a smaller/cheaper M.2 SSD with Linux Mint on it and dual boot to have some fun that way. I think I have an old 500 MB M.2 SSD around somewhere.... Linux is pretty compact.

 

[USAFRet]

Don't get me wrong...NVMe drives are beneficial.
Just not "10X!!" as the ads all say.

And RAID 0 is not 2x as fast as individual drives.


Think of it like this:
Honda Civic and Ferrari F40. Which gets you to the end of your driveway faster?

The F40 benchmarks 3 times faster than the Honda. But you only maybe see that on the rare track day.

 

[dcairns]

Well, considering the amount of data I sling around as I work, I would think there should be some noticeable benefit. But I should learn more first. I would like to get a better handle on bifurcation, if only to learn. Old dogs can still learn, I was upgrading hard drives back when 30MB (yeah Mega bytes) hard drives were amazing

 

[USAFRet]

I was upgrading hard drives back when 30MB (yeah Mega bytes) hard drives were amazing

I was there too.
My 1st HDD was 10MB. Then Stacker software to make it look like 20MB.

 

[thestryker]

Bifurcation is a giant mess for two main reasons: Intel's shoddy desktop support and Broadcom purchasing PLX Technology. When you could get PCIe switches on regular desktop motherboards almost anything ports wise was possible.

If you want to experience first hand the Intel mess Asus has a handy list: https://www.asus.com/support/faq/1037507/

Overall AMD has done a much better job with bifurcation support on desktop platforms.

With regards to noticing performance it depends on what the data sizes are as well using SATA vs NVMe as an example the sequential high queue depth R/W went up to around 7x for PCIe 3.0 and 15x for PCIe 4.0, but low QD random R/W went up by 2-3x and it's still a tiny number. A lot of normal usage is relatively tiny files so it's never taking full advantage. That's why you can easily tell going from HDD to SSD, but moving from SATA to NVMe is much less of a difference. Going from SATA to Optane is typically a much more noticable difference.

Example from CrystalDiskMark when I built my new server:
SK Hynix P41 (MB/s and R/W):
Seq QD32: 7102/6428
RND QD32: 1199/1094
Seq QD1: 5791/5153
RND QD1: 90.06/445.3

Optane P1600X(MB/s and R/W):
Seq QD32: 1793/1071
RND QD32: 1311/1005
Seq QD1: 1653/1033
RND QD1: 356.7/277.2

Keep in mind these are empty drives so as it fills up the NAND drive will slow down slightly. The P41 is 4 lane PCIe 4.0 and the P1600X is 4 lane PCIe 3.0, but the drive design is derived from a 2 lane so it doesn't make use of the extra bandwidth.

 

[Maxxify]

Perhaps I misunderstood the recommendation for the SSD7104/SSD7105 add in cards. I thought their performance boost was due to the RAID 0 striping.

The recommendation was so that you can get 4 drives from one slot. As stated, it's software RAID, so optional. You can run all 4 drives independently. Or you can do just one RAID, with 1-4 drives. Or two RAIDs with 2-4. The total bandwidth is the same.

 

[Maxxify]

Well, considering the amount of data I sling around as I work, I would think there should be some noticeable benefit. But I should learn more first. I would like to get a better handle on bifurcation, if only to learn. Old dogs can still learn, I was upgrading hard drives back when 30MB (yeah Mega bytes) hard drives were amazing

RAID-0 can benefit transfer performance, assuming the source/read drive/RAID is fast enough. There's SLC caching to consider for the write side. Also faster copying. RAID can also help with smaller I/O at very high queue depths, but this is unrealistic. Generally speaking, small I/O latency will be higher if anything in this configuration.

 

[Maxxify]

I am surprised that all these advancements in disk interface speed have not done much for daily use performance. I did expect the usual thing, where benchmarks are optimistic and real life is not as good, but I would have thought you could gain some significant fraction of performance gain.
I think I will return the unopened 4TB M.2 SSD Drive and reconsider my plans here. If my existing 2TB SATA SSD performs as well as the new 4TB M.2 SSD drive, the additional space on the new 4TB SSD is not that big of a deal to me. I routinely move old projects off my working drive to spinning disks for archive.
A better use for the M.2 interface card might be for a smaller/cheaper M.2 SSD with Linux Mint on it and dual boot to have some fun that way. I think I have an old 500 MB M.2 SSD around somewhere.... Linux is pretty compact.

It's largely but not entirely on the software side. More specifically, the storage API. The good news is, DirectStorage is slowly being integrated. Many people think it's for gaming, but it improves storage performance across the stack. The first thing to see this will be applications designed for it with the right workloads, like CAD and some content creation tools. For the time being, you probably only need two or three separate NVMe SSDs to get near top performance out of content creation tools, although RAID can help with larger transfers with the right workflow. More and faster (e.g. DDR5 is > DDR4) RAM is the priority unless CPU-limited.