News Custom PCIe 5.0 SSD with 3D XL-Flash debuts — special Optane-like flash memory delivers up to 3.5 million random IOPS

[Admin]

InnoGrit demonstrates reference solid-state drive based on Kioxia's XL-Flash memory: custom-built XL-Flash-based drives on the way?

Custom PCIe 5.0 SSD with 3D XL-Flash debuts — special Optane-like flash memory delivers up to 3.5 million random IOPS : Read more

 

[bit_user]

Perhaps more importantly, there's less than 13 microseconds of read latency, a huge reduction compared to the ~50 – 100 µs for 3D TLC NAND, and 4 µs write latency, another massive decrease compared to ~200 – 400 µs for 3D TLC NAND

Your range for read latency is skewed a little high. The best PCIe 4.0 drives dipped just below 40 microseconds. I think we've seen even lower, for PCIe 5.0 drives.

Source: https://www.tomshardware.com/reviews/samsung-990-pro-ssd-review/2
​

From the same review, we see write latencies a hair over 10 microseconds:

​

Of course, those writes are to the SLC buffer, not at TLC density. So, that probably explains the 2 orders of magnitude difference. However, there are some TLC drives (IIRC faster Crucial models) which can sustain writes at like several GB/s and I'm not sure they could manage that if their tail latencies were so long.

That said, SCM drives are targeted at enterprise use cases, which often involve sustained writes and require low tail latencies. So, tricks like SLC write-buffering don't really work for them. Instead, this new drive seems to run its entire capacity in SLC mode (which makes it correspondingly more expensive, per bit).

Keep in mind, however, that XL-Flash is a niche type of memory, so it is unlikely to become truly widespread.

Are there any consumer SSDs using it in TLC mode?

 

[jeremyj_83]

Are there any consumer SSDs using it in TLC mode?

The link to the XL-Flash shows that Gen 2 is MLC not TLC.

 

[jeremyj_83]

700k random write IOPS isn't really any better than a mixed use PCIe 5.0 drive like the Micron 9550 MAX. Those have 540k - 720k random write IOPS depending on the size of the drive. However, I guess this might be able to do it at QD4 instead of QD32. Granted they both pale in comparison to the P5800X with 1.5M random write IOPS but only on a PCIe 4 bus.

 

[JRStern]

Well sure, anyone could have built this kind of stuff at any point, but it's all about pricing, so what's the pricing?

 

[jeremyj_83]

Well sure, anyone could have built this kind of stuff at any point, but it's all about pricing, so what's the pricing?

If you have to ask how much it costs you cannot afford it.

 

[bit_user]

700k random write IOPS isn't really any better than a mixed use PCIe 5.0 drive like the Micron 9550 MAX.

I also thought the write IOPS sounded low, but you can't argue with 50 DWPD for 5 years!

 

[Co BIY]

If you have to ask how much it costs you cannot afford it.

Toms could afford to ask for us though!

 

[thestryker]

700k random write IOPS isn't really any better than a mixed use PCIe 5.0 drive like the Micron 9550 MAX. Those have 540k - 720k random write IOPS depending on the size of the drive. However, I guess this might be able to do it at QD4 instead of QD32. Granted they both pale in comparison to the P5800X with 1.5M random write IOPS but only on a PCIe 4 bus.

Gen 1 XL-Flash was fairly close to Gen 2 3D XPoint in performance so I'm wondering if the lower write IOPS here is due to the controller. In SLC mode XL-Flash should absolutely be capable of over 1M IOPS.

 

[thestryker]

I would love to see consumer oriented SSDs using XL-Flash (much like I would have 3D XPoint), but I'm guessing the endurance is endemic as it was for 3D XPoint which makes that virtually impossible to execute. If there's high endurance on the consumer parts everyone will just buy those unless they have specific support contracting. It seems to be the only way to get latencies down and in turn low queue depth performance up.

 

[bit_user]

I would love to see consumer oriented SSDs using XL-Flash (much like I would have 3D XPoint),

Well, in SLC mode, it should be > 3x as expensive per bit as TLC and > 4x as expensive per bit as QLC. That's going to limit the market, quite heavily. Even in MLC mode, it would be > 1.5x as expensive per bit as TLC NAND, yet not as fast as the SLC numbers quoted in the article. Not to mention that max capacity will be down by corresponding amounts.

The part I don't know about is how much overhead the XL part accounts for. The original version touted shorter bitlines and wordlines, as well as more planes. That translates into lower density, while possibly also requiring a more expensive controller that features more channels.

If there's high endurance on the consumer parts everyone will just buy those

Well, the high endurance should mainly come from running it in SLC mode. And if Toshiba is selling XL on the open market, then there's no reason someone couldn't use it in a consumer drive. So, I don't think the lack of such a product is a sign of intentional market segmentation.

People buy datacenter SSDs not just due to the endurance, but the capacity (which is enabled by the form factor), the low tail-latencies, and enterprise-oriented features like out-of-band management, end-to-end data protection, power-loss protection, and more.

Look at it this way: there are datacenter TLC and QLC SSDs that are definitely drawing from the same NAND pools as high-end consumer drives. If you were right that endurance is the only thing that matters, then there's no way those should coexist in the market with consumer TLC and QLC SSDs.

It seems to be the only way to get latencies down and in turn low queue depth performance up.

I'm not sure that SLC benefits read latencies. I think the structure of XL-Flash is probably the main thing responsible for those.

 

[thestryker]

Look at it this way: there are datacenter TLC and QLC SSDs that are definitely drawing from the same NAND pools as high-end consumer drives. If you were right that endurance is the only thing that matters, then there's no way those should coexist in the market with consumer TLC and QLC SSDs.

I think you misunderstood the point I was getting at. Enterprise QLC/TLC drives have significantly better endurance than consumer due to NAND provisioning. I don't think the difference would be anywhere near as stark when using XL-Flash given the inherent endurance benefits.

I'm not sure that SLC benefits read latencies. I think the structure of XL-Flash is probably the main thing responsible for those.

It would be really interesting to see XL-Flash in SLC vs MLC to see what the practical differences are and hopefully we'll see some productized.

 

[Li Ken-un]

When I bought the best Optane to ever be made (P5800X 3.2 TB), I paid around $7.5k for it. I saw it drop to around $4.5k during fire sales when Intel finally put the nail in the coffin. Even with the SLC NAND analog being twice the speed (PCIe 5.0), it’s also got half the endurance and twice the read latency. So I can’t imagine it exceeding $4.5k.

Optane didn’t exist for the raw bandwidth, and it’s still king in all the other areas today.

 

[jeremyj_83]

If there's high endurance on the consumer parts everyone will just buy those unless they have specific support contracting.

You use the enterprise drive because it's guaranteed to work in enterprise software. For example the 905p doesn't have a supported controller in VMware. For your home lab that's fine but not in the data center.

 

[jeremyj_83]

Gen 1 XL-Flash was fairly close to Gen 2 3D XPoint in performance so I'm wondering if the lower write IOPS here is due to the controller. In SLC mode XL-Flash should absolutely be capable of over 1M IOPS.

I haven't found any benchmarks for XL-Flash but based on specs Gen 1 is only around the P4800X in random IOPS. Overall throughout though is higher.

 

[thestryker]

I haven't found any benchmarks for XL-Flash but based on specs Gen 1 is only around the P4800X in random IOPS. Overall throughout though is higher.

There's only two reviews I know of so I've always remembered who did them. Very capacity limited for maximum performance compared to Gen 2 3D XPoint, but it was a much closer competitor than I had expected:
https://www.servethehome.com/dapust...n-slc-optane-alternative-kioxia-intel-optane/

https://www.storagereview.com/review/dapustor-x2900p-scm-ssd-review

 

[bit_user]

I think you misunderstood the point I was getting at. Enterprise QLC/TLC drives have significantly better endurance than consumer due to NAND provisioning.

No, I understood. If that's all there was to datacenter drives, then operators could simply leave some extra, unformatted capacity and call it a day. Once upon a time, like 10+ years ago, SSDs had ways you could explicitly overprovision them, which is something I did on all my SSDs. At some point, FTL became good enough that the advice turned into simply leaving some unformatted capacity. In fact, you don't even need to go that far, if you're good about never filling your drives 100% and regularly using TRIM.

I don't think the difference would be anywhere near as stark when using XL-Flash given the inherent endurance benefits.

What endurance benefits? Where did they ever say the endurance is coming from anywhere other than running it in SLC mode?

It would be really interesting to see XL-Flash in SLC vs MLC to see what the practical differences are and hopefully we'll see some productized.

Agreed. However, if we don't see any examples of it being used in MLC mode, then it's probably safe to assume that it's not particularly competitive against other datacenter SSDs that are using SLC mode.

 

[bit_user]

When I bought the best Optane to ever be made (P5800X 3.2 TB), I paid around $7.5k for it. I saw it drop to around $4.5k during fire sales when Intel finally put the nail in the coffin.

Wow! I only sprang for the 400 GB model. I figured it was most useful to me as a boot drive and 400 GB was big enough for that purpose. It was disheartening to see the ebay price eventually reach about half what I paid for it.

 

[JRStern]

If you have to ask how much it costs you cannot afford it.

If I have to ask how much it costs they can't sell it.
I know, this was just a demonstration, no price available, but even so.
I like the idea.

 

[bit_user]

If I have to ask how much it costs they can't sell it.
I know, this was just a demonstration, no price available, but even so.
I like the idea.

FWIW, here's a link to the product page of their prior XL-Flash SSD.

https://americas.kioxia.com/en-us/business/ssd/enterprise-ssd/fl6.html
​

I searched out one model number, but could find nobody offering it for sale. Even Provantage, which is the one place I know I can find retail (i.e. qty=1) pricing on some datacenter SSDs, who had some Kioxia datacenter SSDs, but not those. They don't carry Dapustor, either (possibly because it's a Chinese brand, I think). Searching specifically for "DapuStor Xlenstor2 X2900P" didn't help, either.

Pricing should be wild, though. Normally, I'd guess a datacenter SSD should cost about 2x the per-TB cost of an enthusiast-class consumer SSD. In this case, I could easily see it running 10x. Partly, due to the lower density, but also because it's a niche/specialty product.

 

[jeremyj_83]

If I have to ask how much it costs they can't sell it.
I know, this was just a demonstration, no price available, but even so.
I like the idea.

The companies that are using a drive like this aren't worried about the cost. A $100k server is pennies to the cost of licensing the software they run on that server.