Question Newly downloaded files on external hard drives are slow to copy/move

[azuree125]

Facing this weird issue where newly downloaded files videos (tv shows, movies) are are slower to copy/move compared files/videos already present on the drive.

I have two 2.5 inch portable HDD's (one seagate onetouch 4tb and another WD passport 5tb) connected to my pc where I keep all my tv shows/movies downloaded. But from the last week or so I'm facing this problem where newly downloaded files are much slower to copy to let say my PC's internal HDD.

Now when I tried to copy those tv shows/videos to another drive the speed would barely touch 15MBps while other similar videos previously downloaded on both drives get copied at upwards of 70MBps.

Also if I make a copy of the video (the one that is slow to copy) on the same external drive (both passport and onetouch) then try to copy/move this “copy” to internal drive this now goes at full speed. I’ve tried drives on other PC’s and got the same behavior there too. And for two separate drives to develop the same issue at the same time is a bit too much of coincidence. I feel there’s something else at play.

Please any help is appreciated.

 

[dwd999]

Since you're talking about HDDs perhaps they need to be defragmented so that new material can be copied to one contiguous space.

 

[azuree125]

Both drives have over one 1TB free space and I've used these drives with as little as 20GB free but never had any such issue. Also this does not explain how after making a copy of the newer files, the "copy" transfers at full speed. I'm honestly stumped at this point.

 

[LinuxDevice]

Maybe this is related, not sure.

Whenever you have a file to move around on a single disk, only the metadata changes and no actual "storage" changes. When you copy from one disk to another, then all of this has to be copied, and this is no longer a simple metadata edit.

Any disk which is present on a typical Windows/Mac/Linux system probably has some internal cache enabled. That cache may not be very big compared to the file size, and tends to help with random access, but is limited. No cache or a cache miss means no advantage from this.

There is also "buffer", and this is much bigger than cache, and is likely what you are experiencing. Buffer is not part of the disk. Most operating systems will use RAM which is not in use as a buffer (if the system needs more RAM, then the buffer is silently discarded and used for whatever else needs it). In order for a buffer to apply the content has to have been accessed once. This buffer will be applied to on the source disk and a separate buffer on the destination disk. If you access that particular disk a second time, then the buffer is read instead of the actual disk. This is a dramatic increase in performance.

If you have never copied (in that boot) a large file from external to internal storage, then you don't have a buffer, but the second time it is read, the buffer will cause the read to be much faster; if you edit that file, then it might slow down again because the buffer has changed and would need to updated.

It really sounds like you are comparing a first operation (with no buffer) to a second operation (which would use buffer and be much faster).

I'll add that if your system does not have enough RAM, then you'll lose buffer. If the RAM is used at the start, then you won't get much buffer. If the RAM is used after you've read from that disk, then the buffer you had will go away until RAM becomes available and you have read from the disk one more time.

Someone else mentioned fragmentation, and if this is an old tech drive with mechanical spinning platters, then that is quite possibly part of the issue. Solid state drives don't really have that issue (and defragmenting a solid state drive would reduce its life without improving anything). A defragmented drive stores content of one file continuously, and once data seek begins, more seek is not generally required; it would just read from start of file to end of file. If the file is fragmented, then it has to seek pieces of the file all over the drive, and seek time is much slower than reading when the head assembly is already where it needs to be. Solid state drives simply have addresses and no physical platter.

Related to what I said about defragmenting a solid state drive lowering its life, a solid state drive has a lot of "cells" for data storage. There is a table of cell addresses which have the information on which address to find a given file (or fragment) at. Every time a cell is written its life decreases, and eventually a cell will fail. This is why writing or editing has hardware and firmware inside of the solid state drive which spreads out which cell is used; each edit will go to a new (unused) cell rather than changing content in one cell, ad this keeps a given cell from wearing out. This is "drive/wear leveling". If a drive is too close to capacity, then wear leveling can slow down by quite a bit. A near-capacity solid state drive is slower than one with some unused reserve. Incidentally, a buffer on top of a solid state drive also increases the life of the drive; changes are accumulated in the buffer prior to flushing to disk, and so there are fewer changes to the actual disk.

 

[azuree125]

Both drives are less than 4 months old. Infact I also tested another 1TB portable WD HDD I had and it showed the exact same problem - newer downloaded files are slower to move and copy compared to the ones already stored on the drive.

 

[Misgar]

Have you considered that all your external drives are probably SMR, not PMR/CMR. This can affect all writes to an SMR drive that already contains a quantity of files. The drive controller has to read back multiple "shingled" tracks, shuffle data around, then write large amounts of data back. Reading from an SMR drive should be relatively fast, but writing to an SMR drive can take ages.

For this reason, when buying internal hard disks, I avoid SMR drives like the plague and only buy PMR/CMR. With external drives, it's difficult to know what you'll get, but SMR crams more data into the same areal area (roughly 5 to 10%) more. My guess is all your external drives are SMR. Hence your problems (when writing to SMR).

SMR drives are really bad news in systems running ZFS such as TrueNAS Core. Resilvering an array of SMR drives can take days, instead of hours with PMR/CMR drives. I would never use SMR drives with ZFS.

SMR is the manufacturer's way of selling you more Gigabytes drive capacity without their having to spend more money. Fine if you're prepared to put up with slow writes on a partially filled drive, but an absolute pain if you're not aware of the "fudge".

Just accept the fact some external hard disks are slow when writing. Alternatively, buy far more expensive portable SSDs.

 

[azuree125]

Even if they are SMR, they working perfectly fine a week ago. And I also don't understand why only the newer files are slower while the older files copy/move at a fast speed. And issue goes away the moment I make a copy of the newer file/video. So for two different drives from two different manufacturers to have the same issue at the same time is very odd.
Also, on checking again I feel one of my internal HDD drive is also kinda showing the same issue but not to the extent of the external drives.

 

[Misgar]

Some internal drives are SMR too. You need to look up the spec to find out what you've got.

I'm still not entirely clear if you're having problems writing files (to possible SMR drives) or reading back files.

In theory, SMR drive should read back just as fast as CMR drives

 

[azuree125]

I don't it has anything to do with SMR/CMR. If that was the case all files stored in the drive would atleast show some slowdown but its only the newer ones that's affected. This is why I'm stumped on where the issue is.

 

[USAFRet]

If this is happening with new files, possibly it is still Indexing.

 

[azuree125]

I downloaded a 1.95GB video yesterday and its still slow to copy. But the moment I make this video's copy, this "copy" transfers at full speed. So I feel its not indexing for so long.

 

[Misgar]

but its only the newer ones that's affected

If the newer files are stored on cylinders close to the centre of the hard disk, reads and writes will occur at half the speed of reads/writes at the outer edge of the hard disk.

When copying, the slowest part of the chain will dictate the maxiumum transfer rate. If you are experiencing slow copies to external/internal hard disks, try the same test on an external/internal SSDs.