Question Why is there no SSD that can do 64GB/s in a full PCIe x16 slot?

YouFilthyHippo

Prominent
BANNED
Oct 15, 2022
216
119
760
Yes, I know there are X16 PCBs you can get that have 4 M2 slots. I don't mean that. I mean a PCIe 5.0 SSD that is itself, one single unit, in an x16 slot that does 64GB/s. Is there one and I just dont know about it?
 
Probably a few reasons:
  • The only market for such would be data centers. There's no practical benefit for consumers to have something like this
    • Not to mention most consumer main boards only have 16 lanes from the CPU itself for the primary PCIe x16 slots. At least until AMD or Intel starts doing HEDT systems again.
  • The 4x M.2 SSDs on one card likely provides better benefits anyway all around
    • The manufacturer doesn't have to make a specialty card
    • The end user can use whatever SSD they want. And reparability becomes slightly better because if one SSD starts to show signs of dying, you only need to replace that, rather than replace the whole thing
 
Yes, I know there are X16 PCBs you can get that have 4 M2 slots. I don't mean that. I mean a PCIe 5.0 SSD that is itself, one single unit, in an x16 slot that does 64GB/s. Is there one and I just dont know about it?
No. You can get some x8 AICs for enterprise, like the Gen4 PM1735, but they don't push the full bandwidth that they could. Most of the time they'll be x4 or x4/x4 (dual-port for redundancy) and may use other form factors like ruler (E1.S/E1.L) which will also be x4 or x8 (or x4/x4). E.3 (E.3S/E.3L) can do up to x16 but usually SSDs will be x4 or x8 (with the possibility for dual-port). In the past there have been some wonky/giant AIC SSDs (e.g. HHHL/FHHL ioDrive which could have multiple PCBs) but at the bandwidth levels you're talking, not that I'm aware of, no.

Multiple reasons for it including the type of controller and amount of flash you would need - consumer tops out at 2400 MT/s channels (3200-3600 eventually with 7nm), even with double the channels as is common with enterprise you wouldn't get there. It's easier to parallelize in other ways (e.g. multiple drives). Flash today can be directly-accessed (SDS) and/or accelerated which can have different form factors including mezzanine, which kind of changes this idea, but is well beyond what you're describing.
 
Last edited:
Question:
From this, and your other similar questions...what is your use case and need for drives like this?

(Not that they exist yet)
I want to do a new build. My last build was in 2019, and even by todays standards, its still better than 90% of PCs on the market today. That's 4 years. It should be ancient, but it isn't. i guess its a testament to how much slower technology is evolving compared to 20 years ago. I want to do a new build, but I want the specs jump to be significant, or theres no point. I'm hoping the Ryzen 9950X will be 24+ cores instead of 16. I also want a big jump in SSD performance. Right now ,the fastest i have seen is 12GB/s on an x4. I dont know why we cant get 48 on an x16 on one single PCB (not a PCB that allows 4 x4 NVMEs). I mean a single PCB in an x16 slot that can do 48GB/s, that is one single storage device, non-RAID
 
Last edited:
I want to do a new build. My last build was in 2019, and even by todays standards, its still better than 90% of PCs on the market today. That's 4 years. It should be ancient, but it isn't. i guess its a testament to how much slower technology is evolving compared to 20 years ago. I want to do a new build, but I want the specs jump to be significant, or theres no point. I'm hoping the Ryzen 9950X will be 24+ cores instead of 16. I also want a big jump in SSD performance. Right now ,the fastest i have seen is 12GB/s on an x4. I dont know why we cant get 48 on an x16 on one single PCB (not a PCB that allows 4 x4 NVMEs). I mean a single PCB in an x16 slot that can do 48GB/s, that is one single storage device, non-RAID
We are deep into diminishing returns on drive speed, and have been for years.

Jumping from HDD to solid state was obviously a huge difference.
From SATA III SSD to NVMe, different, but much smaller benefit.
PCIe 3.0 to 4.0?
Vanishingly small benefit in most use cases.

And on and on.
 
  • Like
Reactions: lmcnabney
Ya. I understand what you're saying. But I feel like jumping from 5GB/s to 64GB/s on PCIe 6.0 would make.a big difference again. Maybe not. I don't know
A 10x boost from SATA III to PCIe 4.0 is nearly unnoticeable.
Similar to 5GBs to 64GBs

Diminishing returns.

10 seconds to 5 seconds....huge difference you can see.

0.1 sec to 0.05 sec....I spent money for what?


In any case, it will be years before we might see that "64GB/s" in the consumer space.
 
Ya. I understand what you're saying. But I feel like jumping from 5GB/s to 64GB/s on PCIe 6.0 would make.a big difference again. Maybe not. I don't know
Only in benchmarks and bragging rights.

Remember that "loading" in a program simply isn't the act of putting data from storage to RAM. It involves initializing the program so it can be used by the user. If you have a PCIe 4.0 SSD, you're at a point now where the storage drive is not the slowest part in the chain for reasonably complicated applications.
 
make.a big difference again
Big difference in what, exactly?

Moving big blocks of data between 2 such drives?
Yes.

Normal consumer PC ops?
No.


Like 2 cars....
Car A has a posted top speed of 190mph.
Car B has a posted top speed of 120mph

Other specs are pretty similar. 40-70mph, 0-60, gas mileage...

Car A costs 3x what Car B costs.
Which one gets you to the end of your driveway faster?
 
  • Like
Reactions: lmcnabney
A 10x boost from SATA III to PCIe 4.0 is nearly unnoticeable.
Similar to 5GBs to 64GBs

Diminishing returns.

10 seconds to 5 seconds....huge difference you can see.

0.1 sec to 0.05 sec....I spent money for what?


In any case, it will be years before we might see that "64GB/s" in the consumer space.
Ok, but why such diminishing returns? I am very confused. Maybe I need an ELI5 on this one. Help?
 
Ok, but why such diminishing returns? I am very confused. Maybe I need an ELI5 on this one. Help?
Lets put some numbers to this then
  • HDD: 100 MB/sec
  • SATA SSD: 500 MB/sec
  • NVMe PCIe 4.0 SSD: 7500 MB/sec
  • PCIe 5.0 x16 SSD: 63,000 MB/sec
If we have something that takes 10 seconds to load from the HDD, then:
  • SATA SSD: 2 seconds to load
  • NVMe SSD: Takes 0.13 seconds to load
  • PCIe 5.0 x16 SSD: Takes 0.016 seconds to load
Are you really going to notice the ~110 ms extra it takes to load something, especially when you consider the amount of time here isn't going to be the total time of the loading process?
 
  • Like
Reactions: lmcnabney
Ok, but why such diminishing returns? I am very confused. Maybe I need an ELI5 on this one. Help?
Per generation, we've had a 2x increase in performance with Sequential data.
THis would be if you were copying a big fat 10GB movie from drive A to drive B.

The small 4k data blocks, where the vast majority of our data lives, does not see that.


SSDs major benefit is the near zero access time. This is across ALL SSD types, even the lowly SATA III drives.


1. The rest of the system comes into play. It is NOT just raw sequential drive speed. CPU, RAM, software, etc.
2. That bog number you see is strictly for marketing purposes. Big numbers sell.


Here are two of my drives, in CrystalDiskMark. PCIe 4.0 980 Pro, and SATA III 860 EVO.
Both 1TB.

CHJYyCM.jpg


The numbers you see advertised in in reviews is the top line.
Big difference, right? The 980 Pro is 10x that of the 860.
But that is only the Sequential speed.

The numbers that really count with the vast majority of our data is the two bottom lines. 4k.
Not much difference, is there?

A few months ago, I did a rendering test with Corels VideoStudio.
Rendering video of 1 min, 5 min, 10 min, out to 3 different drives.
The 980 Pro, Intel 660p, and any of my SATA III SSDs.
Zero difference.
(I should redo this and publish the graphs)

The write speed of the drive made NO difference....the rest of the system has FAR FAR more influence over what happens.


And as said above....you can absolutely tell the difference between 10 seconds and 5 seconds.
You can't tell, outside of artificial benchmarks, the difference between 0.1 sec and 0.05 sec.
Even though it is the same 2x difference.
 
  • Like
Reactions: Roland Of Gilead
Per generation, we've had a 2x increase in performance with Sequential data.
THis would be if you were copying a big fat 10GB movie from drive A to drive B.

The small 4k data blocks, where the vast majority of our data lives, does not see that.


SSDs major benefit is the near zero access time. This is across ALL SSD types, even the lowly SATA III drives.


1. The rest of the system comes into play. It is NOT just raw sequential drive speed. CPU, RAM, software, etc.
2. That bog number you see is strictly for marketing purposes. Big numbers sell.


Here are two of my drives, in CrystalDiskMark. PCIe 4.0 980 Pro, and SATA III 860 EVO.
Both 1TB.

CHJYyCM.jpg


The numbers you see advertised in in reviews is the top line.
Big difference, right? The 980 Pro is 10x that of the 860.
But that is only the Sequential speed.

The numbers that really count with the vast majority of our data is the two bottom lines. 4k.
Not much difference, is there?

A few months ago, I did a rendering test with Corels VideoStudio.
Rendering video of 1 min, 5 min, 10 min, out to 3 different drives.
The 980 Pro, Intel 660p, and any of my SATA III SSDs.
Zero difference.
(I should redo this and publish the graphs)

The write speed of the drive made NO difference....the rest of the system has FAR FAR more influence over what happens.


And as said above....you can absolutely tell the difference between 10 seconds and 5 seconds.
You can't tell, outside of artificial benchmarks, the difference between 0.1 sec and 0.05 sec.
Even though it is the same 2x difference.
Ok, so if the returns now are so bad, whats the point? Why do we keep manufacturing faster SSDs when its clearly not doing anything?
 
Ok, so if the returns now are so bad, whats the point? Why do we keep manufacturing faster SSDs when its clearly not doing anything?
1. New 'standards' keep appearing. PCIe 3.0, 4.0, 5.0, X.0
2. To sell hardware.
3. Because people assume that bigger numbers is a NEED, and glom on to the new shiny. See #2.

Same with ethernet. People assume "Cat 8" is automagically better that 5e because 'bigger number is better!'. In your house, it is not.
 
Ok, but why such diminishing returns? I am very confused. Maybe I need an ELI5 on this one. Help?
tl;dr - you're bottlenecked by software. The biggest gains over HDDs were with random latency and that improves slowly for NAND, plus at a certain point you're bottlenecked by the software being unable to make use of more performance. SSDs work through parallelization which requires queue depth (outstanding I/O requests) but most of your daily apps are QD1 and almost all are QD4 at the most. SSDs are just way too fast and the software isn't optimized for it. At least on Windows and until the DirectStorage API is expanded. Bandwidth is related since BW = 1/latency * size which is more realizable with large I/O but who is transferring around 100GB archives every day?

Enterprise is often focused on latency or long tail performance (worst-case latency/responsiveness) but also will have plenty of queue depth on a cloud server. You don't need that on a glorified YouTube machine.
 
Last edited:
  • Like
Reactions: white.a.drew