Question Newly downloaded files on external hard drives are slow to copy/move

[azuree125]

Facing this weird issue where newly downloaded files videos (tv shows, movies) are are slower to copy/move compared files/videos already present on the drive.

I have two 2.5 inch portable HDD's (one seagate onetouch 4tb and another WD passport 5tb) connected to my pc where I keep all my tv shows/movies downloaded. But from the last week or so I'm facing this problem where newly downloaded files are much slower to copy to let say my PC's internal HDD.

Now when I tried to copy those tv shows/videos to another drive the speed would barely touch 15MBps while other similar videos previously downloaded on both drives get copied at upwards of 70MBps.

Also if I make a copy of the video (the one that is slow to copy) on the same external drive (both passport and onetouch) then try to copy/move this “copy” to internal drive this now goes at full speed. I’ve tried drives on other PC’s and got the same behavior there too. And for two separate drives to develop the same issue at the same time is a bit too much of coincidence. I feel there’s something else at play.

Please any help is appreciated.

 

[dwd999]

Since you're talking about HDDs perhaps they need to be defragmented so that new material can be copied to one contiguous space.

 

[azuree125]

Both drives have over one 1TB free space and I've used these drives with as little as 20GB free but never had any such issue. Also this does not explain how after making a copy of the newer files, the "copy" transfers at full speed. I'm honestly stumped at this point.

 

[LinuxDevice]

Maybe this is related, not sure.

Whenever you have a file to move around on a single disk, only the metadata changes and no actual "storage" changes. When you copy from one disk to another, then all of this has to be copied, and this is no longer a simple metadata edit.

Any disk which is present on a typical Windows/Mac/Linux system probably has some internal cache enabled. That cache may not be very big compared to the file size, and tends to help with random access, but is limited. No cache or a cache miss means no advantage from this.

There is also "buffer", and this is much bigger than cache, and is likely what you are experiencing. Buffer is not part of the disk. Most operating systems will use RAM which is not in use as a buffer (if the system needs more RAM, then the buffer is silently discarded and used for whatever else needs it). In order for a buffer to apply the content has to have been accessed once. This buffer will be applied to on the source disk and a separate buffer on the destination disk. If you access that particular disk a second time, then the buffer is read instead of the actual disk. This is a dramatic increase in performance.

If you have never copied (in that boot) a large file from external to internal storage, then you don't have a buffer, but the second time it is read, the buffer will cause the read to be much faster; if you edit that file, then it might slow down again because the buffer has changed and would need to updated.

It really sounds like you are comparing a first operation (with no buffer) to a second operation (which would use buffer and be much faster).

I'll add that if your system does not have enough RAM, then you'll lose buffer. If the RAM is used at the start, then you won't get much buffer. If the RAM is used after you've read from that disk, then the buffer you had will go away until RAM becomes available and you have read from the disk one more time.

Someone else mentioned fragmentation, and if this is an old tech drive with mechanical spinning platters, then that is quite possibly part of the issue. Solid state drives don't really have that issue (and defragmenting a solid state drive would reduce its life without improving anything). A defragmented drive stores content of one file continuously, and once data seek begins, more seek is not generally required; it would just read from start of file to end of file. If the file is fragmented, then it has to seek pieces of the file all over the drive, and seek time is much slower than reading when the head assembly is already where it needs to be. Solid state drives simply have addresses and no physical platter.

Related to what I said about defragmenting a solid state drive lowering its life, a solid state drive has a lot of "cells" for data storage. There is a table of cell addresses which have the information on which address to find a given file (or fragment) at. Every time a cell is written its life decreases, and eventually a cell will fail. This is why writing or editing has hardware and firmware inside of the solid state drive which spreads out which cell is used; each edit will go to a new (unused) cell rather than changing content in one cell, ad this keeps a given cell from wearing out. This is "drive/wear leveling". If a drive is too close to capacity, then wear leveling can slow down by quite a bit. A near-capacity solid state drive is slower than one with some unused reserve. Incidentally, a buffer on top of a solid state drive also increases the life of the drive; changes are accumulated in the buffer prior to flushing to disk, and so there are fewer changes to the actual disk.

 

[azuree125]

Both drives are less than 4 months old. Infact I also tested another 1TB portable WD HDD I had and it showed the exact same problem - newer downloaded files are slower to move and copy compared to the ones already stored on the drive.

 

[Misgar]

Have you considered that all your external drives are probably SMR, not PMR/CMR. This can affect all writes to an SMR drive that already contains a quantity of files. The drive controller has to read back multiple "shingled" tracks, shuffle data around, then write large amounts of data back. Reading from an SMR drive should be relatively fast, but writing to an SMR drive can take ages.

For this reason, when buying internal hard disks, I avoid SMR drives like the plague and only buy PMR/CMR. With external drives, it's difficult to know what you'll get, but SMR crams more data into the same areal area (roughly 5 to 10%) more. My guess is all your external drives are SMR. Hence your problems (when writing to SMR).

SMR drives are really bad news in systems running ZFS such as TrueNAS Core. Resilvering an array of SMR drives can take days, instead of hours with PMR/CMR drives. I would never use SMR drives with ZFS.

SMR is the manufacturer's way of selling you more Gigabytes drive capacity without their having to spend more money. Fine if you're prepared to put up with slow writes on a partially filled drive, but an absolute pain if you're not aware of the "fudge".

Just accept the fact some external hard disks are slow when writing. Alternatively, buy far more expensive portable SSDs.

 

[azuree125]

Even if they are SMR, they working perfectly fine a week ago. And I also don't understand why only the newer files are slower while the older files copy/move at a fast speed. And issue goes away the moment I make a copy of the newer file/video. So for two different drives from two different manufacturers to have the same issue at the same time is very odd.
Also, on checking again I feel one of my internal HDD drive is also kinda showing the same issue but not to the extent of the external drives.

 

[Misgar]

Some internal drives are SMR too. You need to look up the spec to find out what you've got.

I'm still not entirely clear if you're having problems writing files (to possible SMR drives) or reading back files.

In theory, SMR drive should read back just as fast as CMR drives

 

[azuree125]

I don't it has anything to do with SMR/CMR. If that was the case all files stored in the drive would atleast show some slowdown but its only the newer ones that's affected. This is why I'm stumped on where the issue is.

 

[USAFRet]

If this is happening with new files, possibly it is still Indexing.

 

[azuree125]

I downloaded a 1.95GB video yesterday and its still slow to copy. But the moment I make this video's copy, this "copy" transfers at full speed. So I feel its not indexing for so long.

 

[Misgar]

but its only the newer ones that's affected

If the newer files are stored on cylinders close to the centre of the hard disk, reads and writes will occur at half the speed of reads/writes at the outer edge of the hard disk.

When copying, the slowest part of the chain will dictate the maxiumum transfer rate. If you are experiencing slow copies to external/internal hard disks, try the same test on an external/internal SSDs.

 

[azuree125]

Don't have an SSD to test right now but I personally feel the problem is somewhere else. Files downloaded over a week so ago transfer just fine and I've barely added 20 or 30GB more since then to the drives. SMART data from crystaldiskinfo also show no errors.

 

[Bob.B]

Don't have an SSD to test right now but I personally feel the problem is somewhere else. Files downloaded over a week so ago transfer just fine and I've barely added 20 or 30GB more since then to the drives. SMART data from crystaldiskinfo also show no errors.

When you make a copy of the downloaded file are the file sizes the same?

 

[azuree125]

When you make a copy of the downloaded file are the file sizes the same?

Yep, exact same.

 

[Ralston18]

I suggest using Task Manager, Resource Monitor, and Process Explorer (Microsoft, free) to observe and compare system performance between "slow" and "fast" scenarios.

https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Objective simply being to discover what the system is doing or trying to do with respect to the described transfer rate differences, etc..

Or maybe stops doing: indexing perhaps. What about AV or Malware apps running in the background? Could be some other app being triggered via Task Scheduler getting involved with it all.

Use all three tools but only one tool at a time.

May take a bit of trial and effort to work out how to watch what is happening during attempted copy's and/or move's.

Likely you will need to keep multiple windows open: One window for the file transfer part and another window to observe what the tool presents.

Take your time, be methodical, change (if necessary) only one thing at a time.

What is different between "slow" and "fast"? The tools may reveal something.

 

[azuree125]

I tried what you said and all the data really looked a bit too technical for me to make sense but one thing I noticed in task manager under performance is that when I tried to copy the "original" downloaded videos, the average response time is quite high like near 2000ms and read speed got capped around 22MB/s while when I tried to copy the "copy" of the video which somehow is unaffected, the avg response time was below 1000ms and read speed was over 70MB/s

 

[Bob.B]

I tried what you said and all the data really looked a bit too technical for me to make sense but one thing I noticed in task manager under performance is that when I tried to copy the "original" downloaded videos, the average response time is quite high like near 2000ms and read speed got capped around 22MB/s while when I tried to copy the "copy" of the video which somehow is unaffected, the avg response time was below 1000ms and read speed was over 70MB/s

Is the orig download compressed?

https://windows.tips.net/T005672_Seeing_which_Files_are_Compressed.html

Compare the detail tab between the orig and the copied file.

 

[azuree125]

Is the orig download compressed?

https://windows.tips.net/T005672_Seeing_which_Files_are_Compressed.html

Compare the detail tab between the orig and the copied file.

Now I notice that the attributes are indeed different. The original file has "Attributes : AP" while after making its copy it becomes "Attributes : A". But I see a few other older files with AP attribute which transfers okay. So is this something?

 

[Ralston18]

FYI:

https://answers.microsoft.com/en-us...ai-vs-ap/26decd9c-bc67-432d-92ba-46d85fda78ac

And the link within:

https://learn.microsoft.com/en-us/windows/win32/fileio/file-attribute-constants

Take a look at "P" - Pinned.

"This attribute indicates user intent that the file or directory should be kept fully present locally even when not being actively accessed. This attribute is for use with hierarchical storage management software." In Task Manager what resource(s) were involved with respect to the stated response times and read speeds?

 

[azuree125]

Here's the screenshots from when I'm copying
https://pixhost.to/gallery/wZG1t

 

[Ralston18]

The screenshots clearly indicate that a lot is happening.

I am not sure that I am able to make comparisons between "original" and "copy" file transfers.

The screenshots show that a lot is goind on - especially with respect to the various disk drives.

RAM is also being heavily used.

Are you able to open Disk Management? If so, expand the window so all can be seen, take a screenshot and post that screen shot as well.

Use imgur (www.imgur.com) to post the screenshots.

Noted that Drive J: is a WD MyPassport. How old is that drive? Is the firmware up to date?

 

[azuree125]

The screenshots clearly indicate that a lot is happening.

I am not sure that I am able to make comparisons between "original" and "copy" file transfers.

The screenshots show that a lot is goind on - especially with respect to the various disk drives.

RAM is also being heavily used.

Are you able to open Disk Management? If so, expand the window so all can be seen, take a screenshot and post that screen shot as well.

Use imgur (www.imgur.com) to post the screenshots.

Noted that Drive J: is a WD MyPassport. How old is that drive? Is the firmware up to date?

https://imgur.com/a/4atbqN4

View: https://imgur.com/a/4atbqN4


The drive is barely 4 months old and everything is up to date. And its not just the WD passport, I get the same thing on another Seagate external drive attached.