Question why are two m.2's slower transferring then my download?

fastfed

Distinguished
Dec 7, 2011
29
1
18,535
0
Yes, I basically moved my user name folder from one m.2 to another.
so yea, lots of little files, is this the reason?

Not sure why the system specs have anything to do with it, both are Samsung 970 m.2's
5900x cpu.

If I have a 4790k cpu would it make the difference of 5mgps vs 3000 that the m.2 advertise ?
 

geofelt

Titan
Once a receiving device buffer of the target m.2 gets full, the speed drops because the underlying nand chips need to get updated. This is a read/rewrite process which is very much slower than the read speeds of the source pcie device.
 

USAFRet

Titan
Moderator
Mar 16, 2013
154,632
11,240
176,090
24,153
Yes, I basically moved my user name folder from one m.2 to another.
so yea, lots of little files, is this the reason?

Not sure why the system specs have anything to do with it, both are Samsung 970 m.2's
5900x cpu.

If I have a 4790k cpu would it make the difference of 5mgps vs 3000 that the m.2 advertise ?
Moving 50GB of thousands of small files WILL be slower than moving a single 50GB file.
 
This happens with ALL types of drives.

The CPU and RAM has a major impact in this as well.
It has to locate the next file to work with.
yes, hopefuly win32 storage api gets deprecated soon :)

btw cpu/ram usage is low in case of copying tons of small files, about 2% cpu usage on 8c/16t CPU, on another hand windows defender eats more during file copy ~8% CPU
ya umm which could be botleneck... disable antivirus could help i guess
 

USAFRet

Titan
Moderator
Mar 16, 2013
154,632
11,240
176,090
24,153
yes, hopefuly win32 storage api gets deprecated soon :)

btw cpu/ram usage is low in case of copying tons of small files, about 2% cpu usage on 8c/16t CPU, on another hand windows defender eats more during file copy ~8% CPU
ya umm which could be botleneck... disable antivirus could help i guess
No matter what api, functionality, or OS, thousands of small files transfer slower than a single large file.
Linux is no different than Windows is no different than Apple.

The solid state drives are a LOT faster than spinning, due to the very much reduced access time.
But that time is not "zero".
 
No matter what api, functionality, or OS, thousands of small files transfer slower than a single large file.
Linux is no different than Windows is no different than Apple.

The solid state drives are a LOT faster than spinning, due to the very much reduced access time.
But that time is not "zero".
well ya,, i just tryed on win11 how good it is....1% drive usage on both nvme drives, 2% cpu usage, not really multithreaded process
nvme can handle 65k commands per queue
anyway, writing simple program (5min work max) for copying files from one drive to another while utilising all cpu cores would provide better drive transfers then what windows offers with its explorer lol
 

USAFRet

Titan
Moderator
Mar 16, 2013
154,632
11,240
176,090
24,153
anyway, writing simple program (5min work max) for copying files from one drive to another while utilising all cpu cores would provide better drive transfers then what windows offers with its explorer lol
OK then....please write this, and get it integrated into the Windows ecosystem.
Or even as a standalone application.

You have 5 minutes.
Go.
 
windows doesnt run m2 nvme properly when writing many files, even if drive is capable of simultaneously writing tons of files at once, windows likes to do one file by one...
While one could argue that yes, NVME has provisions for multiple queues and multiple threads for file transfers, the fact of the matter is for every file you copy and write in an OS:
  • The OS has to figure out if you have permission to read the file
  • The OS has to figure out if you have permission to write to the destination
  • The OS has to do some pre-transfer bookkeeping in the file system if it supports such
  • The OS has to figure out where it thinks the best place in the partition is (i.e., which logical block should the data reside in)
  • The storage drive has to figure out where it thinks that data should actually live (logical blocks do not necessarily map cleanly to how things are laid out in the drive)
  • The storage drive has to verify a successful write
  • The OS has to do some book keeping to reflect this in the file system
Take a restaurant for instance, it doesn't matter how many servers you hire, 100 individuals takes longer to service than say 20 groups of 5 or 10 groups of 10 or some other mishmash of groups.
 

ASK THE COMMUNITY