Copying files between hard drives: Why so slow and why the size difference?

[tom2u]

I'm guilty of not finding a good backup solution so I do it traditionally: Erase the folder and replace it with the new folder from the working hard drive. If something goes wrong I can always restore it from the Recycle Bin. But I'm curious why it takes so long. Is it becuase many of the files are small, say about 1 mb each? The total size was 25 gb and took about 10 minutes to copy between 2 7200 rpm Sata drives. Also sometimes the speed drops down a lot. It started at about 50 mb/sec.

 

[Snipergod87]

It takes much longer to read and write alot of smaller files as opposed to a few large files, this has to do with the heads jumping around, Good backup solutions will check to see if any of the data has changed and just copy over the changes greatly reducing the time to make backups and the load on the drives.

 

[ex_bubblehead]

It takes much longer to read and write alot of smaller files as opposed to a few large files, this has to do with the heads jumping around....

There's far more to it than just that. The time required to setup/tear down the I/O channel/s required for the transfer can be a significant, and for small files, the overwhelming factor. This is why you see transfer rates plummet when transferring large numbers of small files.

 

[smashjohn]

The size difference could have to do with how the disks were formatted. If one was formated with 4KB (NTFS) clusters and the other with 128KB (exFAT) clusters, you're going to have different sector resolution. The result is that files take up slightly different amounts of space on drives that are formatted differently.

 

[tom2u]

Great answers. That makes so much sense. Thank you for that clarification. And for that reminder that backup software is the way to go so workload is minimized. I tried some before years ago (can't remember what 3rd party it was) and often found it missed files that I had changed weren't updated in the destination location. After spending so much time checking it was easier to just wipe it and copy it fresh. Any recommendations for good backup software that just updates what has changed?

And does anyone know why the size of all the files is a little different when copied? Shouldn't it be absolutely exact? I've checked the files and they all seem to be there. Is it compressing differently perhaps? Or maybe, as smashjohn mentioned, cluster size differences? But wouldn't the cluster size be identical as that's what's being copied? Or is that affected by the way the drive was partitioned and formatted? (Both drives have 1 NTFS partition.)

 

[Snipergod87]

Depends if you are using compression, or different sector sizes. For my backups I use Veeam Endpoint for Windows Free, it uses Microsofts VSS and the Change Block Transfer Driver to track changes and only copy the differences, though not sure if this would apply to just backing up a few folders vs an entire partition.

 

[Snipergod87]

It takes much longer to read and write alot of smaller files as opposed to a few large files, this has to do with the heads jumping around....

There's far more to it than just that. The time required to setup/tear down the I/O channel/s required for the transfer can be a significant, and for small files, the overwhelming factor. This is why you see transfer rates plummet when transferring large numbers of small files.

Thank's I learned something more!

 

[smashjohn]

And does anyone know why the size of all the files is a little different when copied? Shouldn't it be absolutely exact? I've checked the files and they all seem to be there. Is it compressing differently perhaps? Or maybe, as smashjohn mentioned, cluster size differences? But wouldn't the cluster size be identical as that's what's being copied? Or is that affected by the way the drive was partitioned and formatted? (Both drives have 1 NTFS partition.)

This probably explains it better than I can: http://www.ntfs.com/hard-disk-basics.htm

To see your sector and cluster sizes, run from the command line (replace C: with the drive letter you want to check):
"fsutil fsinfo ntfsinfo c:"