Review HighPoint Rocket 1608A add-in card review: More drives, more power

[Admin]

The HighPoint Rocket 1608A is a new, PCIe 5.0 add-in-card that allows you to boost storage performance with up to eight SSDs in tandem. It’s super fast for the right workloads, if you need that kind of performance.

HighPoint Rocket 1608A add-in card review: More drives, more power : Read more

 

[Amdlova]

60 seconds of speed after that slow as f.
For normal user it's only burn money...
For enterprise user? Nops
For people with red cameras epic win!

 

[thestryker]

Would love to know what the issue with the Intel platforms is as I can't think of any logical reason for it.

Broadcom is the reason why this card is so expensive as they massively inflated costs on PCIe switches after buying PLX Technology. There haven't been any reasonable priced switches at PCIe 3.0+ since.

I have a pair of PCIe 3.0 x4 dual M.2 cards that I imported from China because the ones available in the US were the same generic cards but twice as much money and they were still ~$50 per. Dual slot cards with PCIe switches from western brands are typically $150+ (most of these are x8 with the exception of QNAP who has some x4).

 

[Amdlova]

Would love to know what the issue with the Intel platforms is as I can't think of any logical reason for it.

Broadcom is the reason why this card is so expensive as they massively inflated costs on PCIe switches after buying PLX Technology. There haven't been any reasonable priced switches at PCIe 3.0+ since.

I have a pair of PCIe 3.0 x4 dual M.2 cards that I imported from China because the ones available in the US were the same generic cards but twice as much money and they were still ~$50 per. Dual slot cards with PCIe switches from western brands are typically $150+ (most of these are x8 with the exception of QNAP who has some x4).

Can you share the model?

 

[thestryker]

Can you share the model?

This is the version sold in NA which has come down in price ~$20 since I got mine (uses PCIe 3.0 switch):
https://www.newegg.com/p/17Z-00SW-00037?Item=9SIAMKHK4H6020

You should be able to find versions of on AliExpress for $50-60 and there are also x8 cards in the same price range, just make sure they have the right switch.

The PCIe 2.0 one I got at the time this was the best pricing (you can find both versions for less elsewhere now): https://www.aliexpress.us/item/3256803722315447.html
I didn't need PCIe 3.0 x4 worth of bandwidth because the drives I'm using are x2 which allowed me to save some more money. ASM2812 is the PCIe 3.0 switch and the ASM1812 is the PCIe 2.0 switch.

 

[Amdlova]

@thestryker Thanks

 

[razor512]

60 seconds of speed after that slow as f.
For normal user it's only burn money...
For enterprise user? Nops
For people with red cameras epic win!

Sadly the issue is many SSD makers stopped offering MLC NAND. Consider the massive drop in write speeds after the pSLC cache runs out. Consider even back in the Samsung 970 Pro days.
As long as the SOC was kept cool, the 1TB 970 Pro (a drive from 2018) could maintain its 2500-2600MB/s (2300MB/s for the 512GB drive) write speeds from 0% to 100% fill.

So far we have not really seen a consumer m.2 NVMe SSD have steady state write speeds hitting even 1.8GB/s steady state until they started releasing PCIe 5.0 SSDs, and even then, you are only getting steady state write speeds exceeding that of old MLC drives at 2TB capacity; effectively requiring twice the capacity to achieve those speeds.

The Samsung 990 Pro 2TB drops to 1.4GB/s writes when the pSLC cache runs out.

If anything, I would have liked to see SSD makers continue to produce some MLC drives, since prices have come down, compared to many years ago, a modern 2TB SSD is still cheaper than 1TB drives MLC drives from those days. They could literally make a 1TB drive using faster MLC and charge the price of the 2TB TLC drives, and offer significantly higher steady state write performance for write intensive workloads.

 

[JarredWaltonGPU]

60 seconds of speed after that slow as f.
For normal user it's only burn money...
For enterprise user? Nops
For people with red cameras epic win!

This is largely contingent on the SSDs being used. Samsung 990 Pro 2TB (which is what HighPoint provided) write at up to 6.2 GB/s or so for 25 seconds in Iometer Write Saturation testing, until the pSLC is full. Then they drop to ~1.4 GB/s, but with no "folding state" of lower performance. So best-case eight drives would be able to do 49.6 GB/s burst, and then 11.2 GB/s sustained.

The R1608A doesn't quite hit those speeds, but having software RAID0 and a bit of other overhead is acceptable. I'm a bit bummed that we weren't provided eight Crucial T705 drives, or eight Sabrent Rocket 5 drives, because I suspect either one would have sustained ~30 GB/s in our write saturation test.

Sadly the issue is many SSD makers stopped offering MLC NAND. Consider the massive drop in write speeds after the pSLC cache runs out. Consider even back in the Samsung 970 Pro days. As long as the SOC was kept cool, the 1TB 970 Pro (a drive from 2018) could maintain its 2500-2600MB/s (2300MB/s for the 512GB drive) write speeds from 0% to 100% fill.

So far we have not really seen a consumer m.2 NVMe SSD have steady state write speeds hitting even 1.8GB/s steady state until they started releasing PCIe 5.0 SSDs, and even then, you are only getting steady state write speeds exceeding that of old MLC drives at 2TB capacity; effectively requiring twice the capacity to achieve those speeds.

The Samsung 990 Pro 2TB drops to 1.4GB/s writes when the pSLC cache runs out.

If anything, I would have liked to see SSD makers continue to produce some MLC drives, since prices have come down, compared to many years ago, a modern 2TB SSD is still cheaper than 1TB drives MLC drives from those days. They could literally make a 1TB drive using faster MLC and charge the price of the 2TB TLC drives, and offer significantly higher steady state write performance for write intensive workloads.

You need to look at our many other SSD reviews, where there are tons of drives that sustain way more than 1.4 GB/s. The Samsung 990 Pro simply doesn't compete well with newer drives. Samsung used to be the king of SSDs, and now it's generally just okay. 980/980 Pro were a bit of a fumble, and 990 Pro/Evo didn't really recover.

There are Maxio and Phison-based drives that clearly outperform the 990 Pro in a lot of metrics. Granted, most of the drives that sustain 3 GB/s or more are Phison E26 using the same basic hardware, and most of the others are Phison E18 drives. Here's a chart showing one E26 drive (Rocket 5), plus other E18 drives that broke 3 GB/s sustained, with one 4TB Maxio MAP1602 that did 2.6 GB/s:

 

[abufrejoval]

It's a real shame that now that we have PCIe switches again, which are capable of 48 PCIe v5 speeds and can actually be bought, the mainboards which would allow easy 2x8 bifurcation are suddenly gone... they were still the norm on AM4 boards.

Apart from the price of the AIC, I'd just be perfectly happy to sacrifice 8 lanes of PCIe to storage, in fact that's how I've operated many of my workstations for ages using 8 lanes for smart RAID adapters and 8 for the dGPU.

One thing I still keep wondering about and for which I haven't been able to find an answer: do these PCIe switches effectively switch packets or just lanes? And having a look at the maximum size of the packet buffers along the path may also explain the bandwidth limits between AMD and Intel.

Here is what I mean:

If you put a full complement of PCIe v4 NVMe drives on the AIC, you'd only need 8 lanes of PCIe v5 to manage the bandwidth. But it would require fully buffering the packets that are being switched and negotiating PCIe bandwidths upstream and downstream independently.

And from what I've been reading in the official PCIe specs, full buffering of the relatively small packets is actually the default operational mode in PCIe, so packets could arrive at one speed on its input and leave at another on its output.

Yet what I'm afraid is happening is that lanes and PCIe versions/speeds seem to be negotiated both statically and based on the lowest common denominator. So if you have a v3 NVMe drive, it will only ever have its data delivered at the PCIe v5 slot at v3 speeds, even if within the switch four bundles of four v3 lanes could have been aggregated via packet switching to one bundle of four v5 lanes and thus deliver the data from four v3 drives in the same time slot on a v5 upstream bus.

It the crucial difference between a lane switch and a packet switch and my impression is that a fundamentally packet switch capable hardware is reduced to lane switch performance by conservative bandwidth negotiations, which are made end-point-to-end-point instead of point-to-point.

But I could have gotten it all wrong...

 

[JarredWaltonGPU]

Would love to know what the issue with the Intel platforms is as I can't think of any logical reason for it.

I think it's just something to do with not properly providing the full x16 PCIe 5.0 bandwidth to non-GPU devices? Performance was bascially half of what I got from the AMD systems. And as noted, using non-HEDT hardware in both instances. Threadripper Pro or a newer Xeon (with PCIe 5.0 support) would probably do better.

Here is what I mean:

If you put a full complement of PCIe v4 NVMe drives on the AIC, you'd only need 8 lanes of PCIe v5 to manage the bandwidth. But it would require fully buffering the packets that are being switched and negotiating PCIe bandwidths upstream and downstream independently.

Yeah, that's wrong. There are eight M.2 sockets, each with a 4-lane connection. So you need 32 lanes of PCIe 4.0 bandwidth for full performance... or 16 lanes of PCIe 5.0 offer the same total bandwidth. If you use eight PCIe 5.0 or 4.0 drives, you should be able to hit max burst throughput of ~56 GB/s (assuming 10% overhead for RAID and Broadcom and such). If you only have an x8 PCIe 5.0 link to the AIC, maximum throughput drops to 32 GB/s, and with overhead it would be more like ~28 GB/s.

If you had eight PCIe 3.0 devices, then you could do an x8 5.0 connection and have sufficient bandwidth for the drives.

 

[abufrejoval]

Yeah, that's wrong. There are eight M.2 sockets, each with a 4-lane connection. So you need 32 lanes of PCIe 4.0 bandwidth for full performance... or 16 lanes of PCIe 5.0 offer the same total bandwidth. If you use eight PCIe 5.0 or 4.0 drives, you should be able to hit max burst throughput of ~56 GB/s (assuming 10% overhead for RAID and Broadcom and such). If you only have an x8 PCIe 5.0 link to the AIC, maximum throughput drops to 32 GB/s, and with overhead it would be more like ~28 GB/s.

If you had eight PCIe 3.0 devices, then you could do an x8 5.0 connection and have sufficient bandwidth for the drives.

Well wrong about forgetting that it's actually 8 slots instead of the usual 4 I get on other devices: happy to live with that mistake!

But right, in terms of the aggregation potential would be very nice!

So perhaps the performance difference can be explained by Intel and AMD using different bandwidth negotiation strategies? Intel doing end-to-end lowest common denominator and AMD something better?

HWinfo can usually tell you what exactly is being negotiated and how big the buffers are at each step.

But I guess it doesn't help identifying things when bandwidths are actually constantly being renegotiated for power management...

 

[thestryker]

I think it's just something to do with not properly providing the full x16 PCIe 5.0 bandwidth to non-GPU devices? Performance was bascially half of what I got from the AMD systems. And as noted, using non-HEDT hardware in both instances. Threadripper Pro or a newer Xeon (with PCIe 5.0 support) would probably do better.

Yeah it certainly seems like it's using 8 lanes, but that doesn't really make any sense which would point to an implementation issue either on the card side or client platform. I can't think of any other tests done on an Intel client platform that used more than 8 PCIe 5.0 lanes of bandwidth, but I'd still lean towards it being an issue on the side of the card.

 

[abufrejoval]

Would love to know what the issue with the Intel platforms is as I can't think of any logical reason for it.

It might be a different stategy in terms of bandwidth negotiation: optimal point-to-point vs. lowest common denominator end-to-end. Hopefully the TH guys can find that out, since they now have the hardware.

Broadcom is the reason why this card is so expensive as they massively inflated costs on PCIe switches after buying PLX Technology. There haven't been any reasonable priced switches at PCIe 3.0+ since.

I'd say it's Avago's fault, who now have donned the Broadcom mantle to cover older sins.

I have a pair of PCIe 3.0 x4 dual M.2 cards that I imported from China because the ones available in the US were the same generic cards but twice as much money and they were still ~$50 per. Dual slot cards with PCIe switches from western brands are typically $150+ (most of these are x8 with the exception of QNAP who has some x4).

I got a Sabrent 4-drive M.2 to PCIe x4 v3 adapter mostly because I didn't just want to throw away all these nice 1TB and 2TB Samsung 970 EVO+ drives as I was upgrading to 4TB WD v4 units and looked for the most economic way to reycle them.

With the lower capacity SATA SSDs years before this was easy: 4-8x passive SATA chassis were cheap and then I'd just use a 2nd hand SAS RAID controller to turn them into a cache pool.

But sacrificing 4 PCIe lanes, perhaps even v5 lanes to a just a TB of storage wasn't economical while switch cost, cable extensions or M.2 to U.2 converions also added horrific costs.

It will only use two PCIe v3 lanes for each drive, but since the upstream port is limited to 4 lanes anyway that wan't too much of a loss, even if I'd have preferred full downstream lanes (at the same price ;-)

I also was going to intially use it to get 8TB of NVMe capacity when NVMe v4 SSDs still cost a kidney. But once the WD850 4TB v4 drive came at Samsung Evo+ economy (and those never went to 4TB), that no longer made sense.

At €150 it was an experiment worth making without provable benefits, the HighPoint at 10x is another matter.

I had at one point identified the PCIe switch maker in that device, but I can't remember more than it being an Asian company making two different devices for 8 and 16 PCIe v3 lanes total, but always with a fixed 4:1 up/down ratio at v3 speeds. And it could be the one that's also used in the cheap M.2 RAID NAS that made its round some months ago...

 

[thestryker]

I got a Sabrent 4-drive M.2 to PCIe x4 v3 adapter mostly because I didn't just want to throw away all these nice 1TB and 2TB Samsung 970 EVO+ drives as I was upgrading to 4TB WD v4 units and looked for the most economic way to reycle them.

With the lower capacity SATA SSDs years before this was easy: 4-8x passive SATA chassis were cheap and then I'd just use a 2nd hand SAS RAID controller to turn them into a cache pool.

But sacrificing 4 PCIe lanes, perhaps even v5 lanes to a just a TB of storage wasn't economical while switch cost, cable extensions or M.2 to U.2 converions also added horrific costs.

It will only use two PCIe v3 lanes for each drive, but since the upstream port is limited to 4 lanes anyway that wan't too much of a loss, even if I'd have preferred full downstream lanes (at the same price ;-)

I also was going to intially use it to get 8TB of NVMe capacity when NVMe v4 SSDs still cost a kidney. But once the WD850 4TB v4 drive came at Samsung Evo+ economy (and those never went to 4TB), that no longer made sense.

At €150 it was an experiment worth making without provable benefits, the HighPoint at 10x is another matter.

I had at one point identified the PCIe switch maker in that device, but I can't remember more than it being an Asian company making two different devices for 8 and 16 PCIe v3 lanes total, but always with a fixed 4:1 up/down ratio at v3 speeds. And it could be the one that's also used in the cheap M.2 RAID NAS that made its round some months ago...

ASM2812 only has 8 downstream so that's why the 4 port is limited to 2 lanes per drive: https://www.asmedia.com.tw/product/507Yq4asXFzi9qt4/7c5YQ79xz8urEGr1

ASM2824 is the one with 16, but it's also generally used in 8 lane cards so I'm guessing they chose to save money rather than optimize individual drive throughput potential.

I'd say it's Avago's fault, who now have donned the Broadcom mantle to cover older sins.

Broadcom had already bought PLX Technology and raised the prices before that acquisition happened.

 

[abufrejoval]

What really bothers me about these Broadploit prices is that basically the Ryzen IODs and "chipset" chips are vastly more capable and flexible switch chips selling at far lower prices, which you should be able to cascade to your delight.

For as little as €85 you can even get a six core Ryzen 5 5500 CPU thrown in when you just buy it for the IOD IF/PCIe/USB/SATA switch capabilities.

Perhaps some bored YTber could hack something together, the extra Ryzen should be able to handle quite a bit of DRAM cache, too, even with ECC support or double as a NAS or whatnot.

Against $1500 list price you should be able to do quite a lot, even if perhaps the TDP and slot space might suffer some.

 

[CmdrShepard]

Is this the same HighPoint which had crappy IDE controllers back in the 90s?

 

[abufrejoval]

Is this the same HighPoint which had crappy IDE controllers back in the 90s?

I've used their economy RockedRAID 2680 PCIe x4 SAS controllers (with SATA drives) featuring Marvell chips when I couldn't quite afford to spend more than €1000 on a smart RAID controller.

They still weren't exactly cheap at around €250 and given they were tiny with little more than a single chip. But they did the job reliably for years and only got pushed out when smart RAID controllers became really cheap 2nd hand.

Their prices reflect the low scale of their production but they've always dared to go into niches nobody else cared to fill. I've never had hardware quality issues, if perhaps their software was very basic and not hardened for security.

Most importantly: I never lost data even even across quite a few online expansions and rebuilds. And that's what counts for me.