News Micron shows off world's fastest PCIe 6.0 SSD, hitting 27 GB/s speeds — Astera Labs PCIe 6.0 switch enables impressive sequential reads

[Admin]

Micron has revealed its first PCIe 6.x data center solution, with a paired SSD and Astera Labs network switch sending data bidirectionally at PCIe 6.x speeds.

Micron shows off world's fastest PCIe 6.0 SSD, hitting 27 GB/s speeds — Astera Labs PCIe 6.0 switch enables impressive sequential reads : Read more

 

[usertests]

I like numbers that double.

PCIe 6.x, so-called due to constant revisions and errata to the PCIe 6 spec (we are currently on PCIe 6.3), promises to be the next horizon for enterprise users and, eventually, consumer products.

Do we know if PCIe 6.0 is likely to be relevant to consumers in the near term?

Not whether you need that fast of an SSD or PCIe 6.0 x16 graphics (you don't), just if implementation costs will be high. Some of these PCIe revisions are less painful than others because trace lengths don't have to be reduced. Other than that, we'll see the heat issues all over again.

 

[rluker5]

I wonder if the random read is much faster.

 

[FunSurfer]

Where is all the interesting stuff: What is the power consumption? What is the operating temperature? Does it need external cooler (active or passive)?
After PCIe 5 SSDs high temperatures what is happening with PCIe 6 SSDs?

 

[joartrak]

All for the newer and faster drives BUT we are only now starting to see consumer 8TB models that are PCIe4 and they are pretty expensive. It does feel like we are skipping a few steps. Will we be seeing PCIe8 before we start seeing 16TB at $200-$300?

 

[bit_user]

Today's PCIe 5.0 maxes out at 128 GB/s of bidirectional speeds on an x16 bus. PCIe 6.x doubles this, reaching 256 GB/s over an x16 connection.

Please stop adding together the bandwidth in each direction. It's confusing and really not useful. Most workloads are heavily asymmetric, meaning the bottleneck will be on the uni-directional speed.

The uni-directional speed is what PCIe and frankly most of the industry traditionally quoted. It's really a more recent development that people like Nvidia have begun to spike the numbers by adding together the bandwidth in each direction.

 

[bit_user]

Do we know if PCIe 6.0 is likely to be relevant to consumers in the near term?

I've been wrong before, but I think not. It would further increase motherboard prices, at a time when those are clearly an issue and consumers really don't need more PCIe bandwidth.

I'd bet we'll at least see CXL on consumer platforms before PCIe 6.0. If mainstream CPUs migrate towards using on-package DRAM and CXL becomes the primary memory expansion method, then I could definitely see a use case for PCIe 6.0 / CXL 3.0 on mainstream PC motherboards.

Some of these PCIe revisions are less painful than others because trace lengths don't have to be reduced. Other than that, we'll see the heat issues all over again.

While PCIe 6.0 doesn't increase the clock speed, its PAM4 encoding does require better signal-to-noise ratio. That roughly corresponds to more PCB layers (or maybe more retimers?), which adds cost.

The thing is that I can't say for sure how much margin consumer boards currently have in their S/N ratio, however you can be pretty sure the cheaper boards are close to the limit for PCIe 5.0. And that's where the greatest cost-sensitivity is. I guess they could just limit PCIe 6.0 to premium boards, but then the overall value proposition for even doing it would be less, especially since ThreadRipper and Xeon-W will almost certainly have it.

Another way it adds cost is in the additional die area needed for the PCIe 6.0 controller. PCIe 6.0 adds some new protocol features, so it's not just about PAM4 encoding/decoding they have to accommodate.

 

[bit_user]

I wonder if the random read is much faster.

QD1 @ 4kB? Probably not much. The amount of time to send 4kB over PCIe 5.0 x4 is already just 0.25 usec, which should mean an upper limit of 4M IOPS.

Sadly, I'm not having much luck finding reviews of high-end enterprise SSDs that actually test QD1, because that's just not an important use case for them. However, I think you can probably look at high-end consumer SSD benchmarks to get a sense of just how far off we are from being bandwidth-limited on 4k random read @ QD1.

In fact, this high-end PCIe 5.0 SSD from last year quotes 4k random read (probably at something like QD256) at just 2.8M.

https://www.storagereview.com/revie...ance-storage-dapustor-haishen5-h5100-e3-s-ssd
​

Since I'm reduced to quoting manufacturer specs, I guess I should also mention that Micron's 9550 (from about the same time) quotes 2.8M to 3.3M 4k random read IOPS. Okay, at 3.3M, we're getting close enough to link saturation that I'd definitely expect to see a benefit from higher speeds.

https://www.storagereview.com/news/micron-9550-ssds-feature-speed-and-power-efficiency
​

But, I maintain that shaving off like 0.13 microseconds from each read isn't going to do much for your QD1 case. Random 1MB reads would be a different story, though.

 

[bit_user]

Where is all the interesting stuff: What is the power consumption?

Lots.

What is the operating temperature?

Hot.

So, they won't yet have detailed specs on these drives (not that I've seen, anyway), but you can get a rough idea of what we're talking about by looking at their previous-gen perf-optimized enterprise SSDs. The datasheet for the 9550 says it averages 18W for peak sequential read and 16W for peak sequential write.

https://www.micron.com/content/dam/...product-flyer/9550-nvme-ssd-product-brief.pdf
​

Supported operating temperatures are up to 70C.

Does it need external cooler (active or passive)?

It's for servers, which have optimized airflow that will be routed through the storage array. Themselves, these drives don't contain an active cooling element, but only because they can offload this requirement onto the server chassis.

Not sure if you've ever been inside a modern server, but they even have hot-swap fans!

After PCIe 5 SSDs high temperatures what is happening with PCIe 6 SSDs?

Intel added PCIe throttling to the things their thermal driver can do to limit CPU overheat conditions.

SSDs have built-in thermal throttling for a long time, but I don't know if that includes dropping back to a lower PCIe rate. They will do this to save power, when you have ASPM enabled. Consumer ones, at least.

 

[thestryker]

I wonder if the random read is much faster.

Nope, and it won't be unless you're using a SCM type drive (even then PCIe revision will have no real impact on this performance). With Intel shutting down Optane more SCM drives have cropped up from the major vendors for enterprise, but they all seem pretty focused on specific workloads rather than being universally better. Nothing I've seen indicates anything of the sort will be headed to consumer grade drives, and this is likely due to limited cooling and board space.

edit: AFAIK this is the best NAND SSD with regards to low latency/random performance and they haven't made a new version (though this may be due to Kioxia and how they've handled XL-Flash): https://www.storagereview.com/review/dapustor-x2900p-scm-ssd-review

edit2: these are the SCM class drives out today:
https://www.micron.com/products/storage/ssd/data-center-ssd/xtr-ssd
https://www.solidigm.com/products/data-center/d7/p5810.html (StorageReview)

 

[Tanquen]

Cool, I'm guessing it still slows down to a few hundred k per second when copying large number of small files? I'm sure it'll be fun copying 30 GB virtual discs for my VMs, but that's about it.

 

[bit_user]

Nope, and it won't be unless you're using a SCM type drive (even then PCIe revision will have no real impact on this performance). With Intel shutting down Optane more SCM drives have cropped up from the major vendors for enterprise, but they all seem pretty focused on specific workloads rather than being universally better. Nothing I've seen indicates anything of the sort will be headed to consumer grade drives, and this is likely due to limited cooling and board space.

edit: AFAIK this is the best NAND SSD with regards to low latency/random performance and they haven't made a new version (though this may be due to Kioxia and how they've handled XL-Flash): https://www.storagereview.com/review/dapustor-x2900p-scm-ssd-review

@rluker5 didn't say whether he meant QD1, 4k, or neither. If we look at QDn 4k random IOPS, the drive you linked claims only 1.8M and 1.2M for reads and writes, respectively. In post #8 I already linked a newer Dapustore model that promises 2.8M and 0.38M reads and writes. Furthermore, the Micron 9550 SSD I also linked in that post claims up to 3.3M and 0.9M reads and writes, respectively.

If @rluker5 said writes, you'd have a point. Since he just specified reads, the drive you cited and the Optane NVMe drives have already been surpassed by NAND-based drives, in terms of QDn IOPS.

edit2: these are the SCM class drives out today:
https://www.micron.com/products/storage/ssd/data-center-ssd/xtr-ssd

In terms of IOPS, it's nothing special. 0.9M and 0.25M for read and write. Also, it's a PCIe 4.0 drive.

https://www.solidigm.com/products/data-center/d7/p5810.html (StorageReview)

P5810 is old news. That review is dated Nov. 9, 2023. Read/Write IOPS: 0.865M/0.495M.

 

[bit_user]

Cool, I'm guessing it still slows down to a few hundred k per second when copying large number of small files?

That's Windows' fault. Try another OS.

https://superuser.com/questions/1124472/why-is-linux-30x-faster-than-windows-10-in-copying-files
​

 

[thestryker]

@rluker5 didn't say whether he meant QD1, 4k, or neither. If we look at QDn 4k random IOPS, the drive you linked claims only 1.8M and 1.2M for reads and writes, respectively. In post #8 I already linked a newer Dapustore model that promises 2.8M and 0.38M reads and writes. Furthermore, the Micron 9550 SSD I also linked in that post claims up to 3.3M and 0.9M reads and writes, respectively.

If @rluker5 said writes, you'd have a point. Since he just specified reads, the drive you cited and the Optane NVMe drives have already been surpassed by NAND-based drives, in terms of QDn IOPS.


In terms of IOPS, it's nothing special. 0.9M and 0.25M for read and write. Also, it's a PCIe 4.0 drive.


P5810 is old news. That review is dated Nov. 9, 2023. Read/Write IOPS: 0.865M/0.495M.

I assumed low QD workloads because otherwise they've always scaled with interface speed so there wouldn't be a question whether it'd be improved or not.