[SOLVED] Are NVMe SSD speeds a myth

Nov 4, 2021
6
0
10
We're trying to achieve this:

An external 4TB USB SSD that can sustain >870 MB/s sequential write of 2.5TB, in real-world copy of multi-GB files in Windows10, on Threadripper PCs.

Spit in any direction, and you'll find an NVMe SSD that does >2000MB/s. And we easily get 900-1050MB/s in synthetic tests (benchmark utilities). But the above? Me and 3 other technicians have spent a week hacking at it... to no avail. I didn't think this benchmark would be such a holy grail quest.

Our 3 test systems:
2 x Gigabyte TRX40 Desginare, 3960X Threadripper
1 x Gigabyte TRX40 Aorus Master, 3970X Threadripper
1 x ASRock TRX40 Creator, 3960X Threadripper
64GB of RAM or better
(In 2 different cities, so I doubt it's old burial ground voodo causing it.)

Here's an example.
Reading from an internal 4TB Sabrent M.2 NVMe SSD, USB-C 10Gbps mobo port, to a 10Gb USB-C enclosure
1.9TB dataset, average file size 3.3GB.
2022-01-29-z-Edit114.png


You can see:
Part 1: Probably the Windows write cache, since that ran at 1.5GB/s, considerably faster than the USB link speed.
Part 2: ~780MB/s, better than 5Gbps speeds...
Part 3: Dropped to a crappy 260MB/s, 20% / 8 minutes / 380GB into the copy

So far we've tried:
  • Sabrent 4TB Rocket NVMe (SB-ROCKET-4TB) (QLC)
  • Sabrent 4TB Rocket 4 Plus NVMe 4.0 Gen4 ("R/W 7100/6600MB/s") (TLC)
  • PNY XLR8 CS3040 4TB NVMe Gen4x4 SSD (TLC)
  • Sabrent 10Gbps USB enclosure
  • Orico 10Gbps USB enclosure
  • Orico 20Gbps USB enclosure with extra heat-sync and active cooling fan
  • Changing the Windows device properties to "High performance", and trying different Write cache settings.
  • Reformat as GPT / NTFS
  • 64KB, 512KB, and 1024KB block size
...no dice.

Sometimes we get sustained 600MB/s, sometimes we get these:
2022-01-28-Pasadena-2.png
2022-01-28-Pasadena.png


As said in the beginning, speed tests aren't of much use:
Pasadena-synthetic-tests.png



PS If we could get that performance when copying thousands of 30MB files, that'd be great too. Not sure if that size still counts as "large files, sequential" vs. "high-IOPS", hence it might warrant a separate discussion. And I'm focusing on multi-GB files first.
 
Solution
It's not a myth, but advertising is the key thing here.

The large benchmark numbers are for a single sequential file.
Copying 30k small files will be much slower.

Like 2 cars, other wise identical, but one has a benchmark top speed of 140mph and the other has a benchmark top speed of 170mph.
If those are the only numbers you see, you'll surmise the 170mph car is faster.
The 0-60mph times are identical to within 0.1 sec.
And that 0-60 is all you and I ever do.


Copying a large single file between 2x fast NVMe shows big speed.
Copying that same file from a fast NVMe to anything else....the slowest device in the chain is the issue.

USAFRet

Titan
Moderator
It's not a myth, but advertising is the key thing here.

The large benchmark numbers are for a single sequential file.
Copying 30k small files will be much slower.

Like 2 cars, other wise identical, but one has a benchmark top speed of 140mph and the other has a benchmark top speed of 170mph.
If those are the only numbers you see, you'll surmise the 170mph car is faster.
The 0-60mph times are identical to within 0.1 sec.
And that 0-60 is all you and I ever do.


Copying a large single file between 2x fast NVMe shows big speed.
Copying that same file from a fast NVMe to anything else....the slowest device in the chain is the issue.
 
Solution
IDK this afternoon I had to move some of my Macrium backup image files from my Sabrent EC-SNVE (containing a 1TB Samsung 970 EVO Plus) onto my computer's internal 1TB WD SN850 and I got sustained speeds around 870 MB/s from external to internal and 815 MB/s from internal to external. If your speeds are dropping off so much it sounds like you've got an overheating problem with one of the components.
 
Nov 4, 2021
6
0
10
Is there a benchmark utility that could simulate lengthy loads better? For example, continuously write 1 TB, or continuously read that much. To test one component at a time, as you said dwd999, against overheating.
 

USAFRet

Titan
Moderator
Is there a benchmark utility that could simulate lengthy loads better? For example, continuously write 1 TB, or continuously read that much. To test one component at a time, as you said dwd999, against overheating.
Find a couple hundred GB data set, and copy into the relevant drive. Monitor the temps.
Doesn't matter what....games, videos, whatever. This is just a copy, so you'll be deleting after.