Question File Copying Speed

Toggle sidebar Toggle sidebar
R

RickVS

Distinguished
Jul 27, 2014
70
0
18,530
I have a USB 3.1 gen 2 hub and a USB 3.1 gen 2 SSD. My optimum transfer speed is around 300 Mb/s. However quite often the speed is a tenth of that, or sometimes it starts at around a tenth of that for the first 25% of the transfer then ratchets up to 300 Mb/s for the rest of the transfer. Most of the time my fast USB and SSD are disappointing in this regards. It's been going on forever, though I thought I'd just now ask to see if I could 1) get an explanation and 2) see if there is anything I can do to fix it and get reliable speeds all of the time. I'm copying from an internal HDD. Thanks.
 
Update your post to include full system hardware specs and OS information.

Regarding "internal HDD" - make, model, capacity, how full?

Try using Task Manager and Resource Monitor to observe system performance when transferring files.

Use both tools but only one tool at a time.

Knowing what system resources are being used, to what extent (%), and what is using any given resource may help identify why the transfer speeds vary and change.

Process Explorer (Microsoft, free) may prove helpful as well.

https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Do some methodical testing to determine what is happening. When speeds are slow, fast, or suddenly changing.

For example, some other background app could be launching for whatever reason(s). Or Task Scheduler is being triggered and launching another app/process.

Key is to discover when and why the speeds change for better or worse.
 
RickVS said:
I have a USB 3.1 gen 2 hub and a USB 3.1 gen 2 SSD. My optimum transfer speed is around 300 Mb/s. However quite often the speed is a tenth of that, or sometimes it starts at around a tenth of that for the first 25% of the transfer then ratchets up to 300 Mb/s for the rest of the transfer. Most of the time my fast USB and SSD are disappointing in this regards. It's been going on forever, though I thought I'd just now ask to see if I could 1) get an explanation and 2) see if there is anything I can do to fix it and get reliable speeds all of the time. I'm copying from an internal HDD. Thanks.
Click to expand...
Performance depends on the slowest device in the chain.

HDD
HDD cache
Full SSD cache
Multiple levels of USB interface
Composition of the actual data
etc
etc
etc
 
  • Like
Reactions: geofelt
First issue would be which interface is the slowest. That's going to dictate the maximum throughput.

Then there's the matter of the OS interacting with the file system in order to make sure the user has permission to do the operation, finding a spot for it in the physical storage, etc. Every file you have, has some level of overhead regarding this. It takes longer to copy 1,000 files totaling 5 GB than it does a single 5GB file.

Then there's cache on the drive. Writes to a storage drive often start off stupid fast because of the cache within them. This is often to make smaller writes (which happen more often) appear to be done sooner.

If you want to make sure your storage is working "as intended", copy a really large file (like 5+GB) over. If the file more or less transfers in the amount of time you expect, then there's nothing wrong.
 
When copying 50GB MP4 video files from NVMe to hard disk, I see reported write speeds starting at 2.5GB/s for 2 to 3 seconds as the hard disk cache fills up. For the remainder of the copying process, write speeds drop to 125GB/s, i.e. the native speed of the drive. There's always a limit somewhere. It's a question of knowing what it is.

Regardless of interface, USB, SATA, SAS, SCSI, etc,, copying small files from one drive to another takes longer than large files, on a per Gigabyte basis. As hitaru.hino said, there's the overhead caused by the operating system, when it decides where to write each file.

When writing files to LTO4 tape, my tape drive needs to be fed with a constant 80MB/s. When backing up thousands of small files (less than 10MB each), even a Gen4 M.2 NVMe drive cannot maintain a constant 80MB/s. The milliseconds taken by Windows each tme it looks for the next file results in unacceptable delays and the tape drive "shoe shines".

For this reason I usually zip or rar complete directories of files, before sending them to tape. The "sweet spot" seems to be 15MB. Anything smaller and the tape drive stops, rewinds and restarts (shoe shines).

I you want to maximise transmission speeds to your SSD, consider zipping all your small files into one large file first, before copying via USB. You won't save any time, if fact it will probably take longer, but your write speeds should increase to 300MB/s, or whatever the limit is on your SSD or USB inteface.

To sum up:
small files = slow
large files = fast
 
The operation of the target ssd is probably in play here.
Some will be better than others.
Initially, any sort of a cache can handle writes quickly.
Then, there will be a fair number of available nand blocks available to receive writes without needing to do a read/rewrite operation.
Then as the availability of free blocks diminishes, the ssd takes longer to write.
Eventually the ssd needs to do major work to keep things going.

SSD devices are great for fast reads.
But, I think backups to a HDD would be a better choice.
 
Ralston18 said:
Update your post to include full system hardware specs and OS information.

Regarding "internal HDD" - make, model, capacity, how full?

Try using Task Manager and Resource Monitor to observe system performance when transferring files.

Use both tools but only one tool at a time.

Knowing what system resources are being used, to what extent (%), and what is using any given resource may help identify why the transfer speeds vary and change.

Process Explorer (Microsoft, free) may prove helpful as well.

https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Do some methodical testing to determine what is happening. When speeds are slow, fast, or suddenly changing.

For example, some other background app could be launching for whatever reason(s). Or Task Scheduler is being triggered and launching another app/process.

Key is to discover when and why the speeds change for better or worse.
Click to expand...
Thanks. 1) Win 10 - i7 6th gen - 32 Gb RAM - 1 Tb SSD (the drive that I copy FROM) - 4 & 6 Tb HDDs that really aren't involved - 500 Gb external SSD (that I copy TO). When I transfer files the CPU floats around under 20% while the RAM is 48 % used with disk utilization very minimal. Nothing is happening on the PC when the speed changes.
 
hotaru.hino said:
First issue would be which interface is the slowest. That's going to dictate the maximum throughput.

Then there's the matter of the OS interacting with the file system in order to make sure the user has permission to do the operation, finding a spot for it in the physical storage, etc. Every file you have, has some level of overhead regarding this. It takes longer to copy 1,000 files totaling 5 GB than it does a single 5GB file.

Then there's cache on the drive. Writes to a storage drive often start off stupid fast because of the cache within them. This is often to make smaller writes (which happen more often) appear to be done sooner.

If you want to make sure your storage is working "as intended", copy a really large file (like 5+GB) over. If the file more or less transfers in the amount of time you expect, then there's nothing wrong.
Click to expand...
The transfers actually often start stupid slow then speed up a quarter the way in. I just tried copying a 5 Gb file and it was $100Mb/s which is a third of what I am able to ofter get in terms of speed.
 
geofelt said:
The operation of the target ssd is probably in play here.
Some will be better than others.
Initially, any sort of a cache can handle writes quickly.
Then, there will be a fair number of available nand blocks available to receive writes without needing to do a read/rewrite operation.
Then as the availability of free blocks diminishes, the ssd takes longer to write.
Eventually the ssd needs to do major work to keep things going.

SSD devices are great for fast reads.
But, I think backups to a HDD would be a better choice.
Click to expand...
The media files are copied from C: SSD to the USB 3.1 gen 2 SSD. Not a backup. Just a transfer to another medium for viewing.
 
Misgar said:
When copying 50GB MP4 video files from NVMe to hard disk, I see reported write speeds starting at 2.5GB/s for 2 to 3 seconds as the hard disk cache fills up. For the remainder of the copying process, write speeds drop to 125GB/s, i.e. the native speed of the drive. There's always a limit somewhere. It's a question of knowing what it is.

Regardless of interface, USB, SATA, SAS, SCSI, etc,, copying small files from one drive to another takes longer than large files, on a per Gigabyte basis. As hitaru.hino said, there's the overhead caused by the operating system, when it decides where to write each file.

When writing files to LTO4 tape, my tape drive needs to be fed with a constant 80MB/s. When backing up thousands of small files (less than 10MB each), even a Gen4 M.2 NVMe drive cannot maintain a constant 80MB/s. The milliseconds taken by Windows each tme it looks for the next file results in unacceptable delays and the tape drive "shoe shines".

For this reason I usually zip or rar complete directories of files, before sending them to tape. The "sweet spot" seems to be 15MB. Anything smaller and the tape drive stops, rewinds and restarts (shoe shines).

I you want to maximise transmission speeds to your SSD, consider zipping all your small files into one large file first, before copying via USB. You won't save any time, if fact it will probably take longer, but your write speeds should increase to 300MB/s, or whatever the limit is on your SSD or USB inteface.

To sum up:
small files = slow
large files = fast
Click to expand...
These are mostly movie files that I've ripped from DVD with the intent of the external SSD drive plugging into my smart TV. So probably doesn't make sense to zip everything. Plus given that a portion of my attempts are actually fast, I know the system is capable of it. And it keeps giving me different speeds even when there is literally nothing happening on the computer outside of Windows sleeping in the background.
 
Since you seem to be transferring large files, the small file "overhead" shouldn't exist. Similarly, your external drive is an SSD, so access times should be faster than a heavily fragmented 2.5" hard disk. Do you know if TRIM is enabled and working on the external SSD? If the USB drive interface supports UASP then TRIM will probably work. The SSD should also incorporate its own disk housekeeping, but it might slow down if either/both of these options are disabled.
 
Does the external SSD have its' own power supply or is the external SSD now using power provided via the computer's USB port?

Try using a independently powered USB hub.

Connect the USB hub to the computer's USB port and then connect the external SSD to the independently powered hub.

The USB hubs are generally inexpensive but try to borrow one for testing purposes.

Keep it simple. No need for a hub with switches or more than 3 or 4 ports.
 
Ralston18 said:
Does the external SSD have its' own power supply or is the external SSD now using power provided via the computer's USB port?

Try using a independently powered USB hub.

Connect the USB hub to the computer's USB port and then connect the external SSD to the independently powered hub.

The USB hubs are generally inexpensive but try to borrow one for testing purposes.

Keep it simple. No need for a hub with switches or more than 3 or 4 ports.
Click to expand...
Thanks. So the external SSD is basically a large 500 Gb flash drive. I also copy at times to a 128 Gb flash drive with similar results. They are both gen 2. They are both powered by the USB 3.1 gen 2 USB hub which is separately powered. The hub is plugged into a gen 2 USB port in my computer.
 
Thanks. I'm actually copying one 5 Gb file. The transfer speeds are 1/10th what they should be (20 ish mb/s). I've also tried this on two different gen 2 hubs each plugged into a different gen 2 usb port on my PC.
 
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom