[SOLVED] Are NVMe SSD speeds a myth

[binba]

We're trying to achieve this:

An external 4TB USB SSD that can sustain >870 MB/s sequential write of 2.5TB, in real-world copy of multi-GB files in Windows10, on Threadripper PCs.

Spit in any direction, and you'll find an NVMe SSD that does >2000MB/s. And we easily get 900-1050MB/s in synthetic tests (benchmark utilities). But the above? Me and 3 other technicians have spent a week hacking at it... to no avail. I didn't think this benchmark would be such a holy grail quest.

Our 3 test systems:
2 x Gigabyte TRX40 Desginare, 3960X Threadripper
1 x Gigabyte TRX40 Aorus Master, 3970X Threadripper
1 x ASRock TRX40 Creator, 3960X Threadripper
64GB of RAM or better
(In 2 different cities, so I doubt it's old burial ground voodo causing it.)

Here's an example.
Reading from an internal 4TB Sabrent M.2 NVMe SSD, USB-C 10Gbps mobo port, to a 10Gb USB-C enclosure
1.9TB dataset, average file size 3.3GB.

You can see:
Part 1: Probably the Windows write cache, since that ran at 1.5GB/s, considerably faster than the USB link speed.
Part 2: ~780MB/s, better than 5Gbps speeds...
Part 3: Dropped to a crappy 260MB/s, 20% / 8 minutes / 380GB into the copy

So far we've tried:

• Sabrent 4TB Rocket NVMe (SB-ROCKET-4TB) (QLC)
• Sabrent 4TB Rocket 4 Plus NVMe 4.0 Gen4 ("R/W 7100/6600MB/s") (TLC)
• PNY XLR8 CS3040 4TB NVMe Gen4x4 SSD (TLC)
• Sabrent 10Gbps USB enclosure
• Orico 10Gbps USB enclosure
• Orico 20Gbps USB enclosure with extra heat-sync and active cooling fan
• Changing the Windows device properties to "High performance", and trying different Write cache settings.
• Reformat as GPT / NTFS
• 64KB, 512KB, and 1024KB block size

...no dice.

Sometimes we get sustained 600MB/s, sometimes we get these:

As said in the beginning, speed tests aren't of much use:

PS If we could get that performance when copying thousands of 30MB files, that'd be great too. Not sure if that size still counts as "large files, sequential" vs. "high-IOPS", hence it might warrant a separate discussion. And I'm focusing on multi-GB files first.

 

[USAFRet]

It's not a myth, but advertising is the key thing here.

The large benchmark numbers are for a single sequential file.
Copying 30k small files will be much slower.

Like 2 cars, other wise identical, but one has a benchmark top speed of 140mph and the other has a benchmark top speed of 170mph.
If those are the only numbers you see, you'll surmise the 170mph car is faster.
The 0-60mph times are identical to within 0.1 sec.
And that 0-60 is all you and I ever do.


Copying a large single file between 2x fast NVMe shows big speed.
Copying that same file from a fast NVMe to anything else....the slowest device in the chain is the issue.

 

[dwd999]

IDK this afternoon I had to move some of my Macrium backup image files from my Sabrent EC-SNVE (containing a 1TB Samsung 970 EVO Plus) onto my computer's internal 1TB WD SN850 and I got sustained speeds around 870 MB/s from external to internal and 815 MB/s from internal to external. If your speeds are dropping off so much it sounds like you've got an overheating problem with one of the components.

 

[binba]

Is there a benchmark utility that could simulate lengthy loads better? For example, continuously write 1 TB, or continuously read that much. To test one component at a time, as you said dwd999, against overheating.

 

[USAFRet]

Is there a benchmark utility that could simulate lengthy loads better? For example, continuously write 1 TB, or continuously read that much. To test one component at a time, as you said dwd999, against overheating.

Find a couple hundred GB data set, and copy into the relevant drive. Monitor the temps.
Doesn't matter what....games, videos, whatever. This is just a copy, so you'll be deleting after.