Linux USB HDD Drives failing

[mirkog]

Hello, I have a strange problem with any new (probably any) HDD I buy.

Here is my hardware and software:
Laptop: Thinkpad T470
Main OS: Ubuntu MATE 16.04.02
Rarely Used OS: Windows 10

Since my old disk is failing (320GB) with bad sectors, I decided to buy a new hybrid one (Seagate ST1000LX015) with 1TB . After I got it I put it into a JMicron external case so I can use it with my laptop.
I first run some benchmarks with Gnome Disks app and everything was fine with a speed of 140 MB/s max and 100 MB/s everage.
When I started transferring the data from the failing disk to the new one it worked ok for the first ~150 GB, by that time it was running very slow (was monitoring speed using the iotop command since most file managers dont handle well the linux caching feature like windows)

I can confirm that the old failing disk was not the problem since it was not blinking (idle) and I was able to use its filesystem, but the new disk was blocked, it was blinking all the time and could not access the filesystem.
I didnt know that the SMART data could be used even through USB so I could just feel the drive was very hot (the drive case was of metal)
Also every time I plugged the new drive on it would start clicking/working all the time even if there was no transfer active (also led didn't blink)
Also I have a desktop PC to which I connected the drive directly to SATA and smart data showed a large number of read errors
I thought it was a hardware (manufacturing) issue so I returned it and got a new one

When I got the replacement disk (also Seagate ST1000LX015) I also bought another case (plastic this time) but got the same result, when I went to return the disk then tested the disk and it worked ok so I got my money back (I did not return the case since it was working great with my old disk (the old one))

After all this I learned that the smart data can be accessed through USB by using the smartctl command

so this time I bought a 1TB external disk (Adata HV620 or AHV620-1TU3-CBK)
I tested the disk using gnome disks as the average read rate was ~100MB/s and write ~70MB/s with no major fluctuations
The SMART data showed no read errors
Then I started copying the same data and again after ~150GB transfered the new disk was extremely slow and constantly blinking and unresponsive while the old failing drive was responsive and idle... and on about 30°C
The new drive was on 40°C

After all those speed issues it managed to transfer the files and in the end the new drive was on 46°C while the old one was on 33°C
After this transfer I used the sync command to flush all the data and tested the speeds with gnome disks and it reported very low write speeds even under 1MB/s with read speed of max 70MB/s with major fluctuations

and again the disk was constantly clicking/active without any transfer (led not blinking)

after a few hours of letting it do the clicking noises the speeds started to improve.

Please help, I dont know what to do anymore since every disk I try starts failing, is it some Linux issue with new drives or maybe a Thinkpad laptop issue?

If you need any other information please let me know.

 

[Mark RM]

are you powering these drives externally? or just using USB power?

 

[mirkog]

are you powering these drives externally? or just using USB power?

Just USB power

 

[Mark RM]

I suspect you're not getting all the power the drives need from the USB subsystem and this is the cause for your problem. You can test this theory several ways
- Get a USB C to USB3 adapter and plug the drive into that making more power available
- plug a powered hub into the laptop and plug the drive into that (I prefer this test).

I suspect this because your returned drive tested OK. It's only when attached to your laptop you have issues.

 

[mirkog]

I suspect you're not getting all the power the drives need from the USB subsystem and this is the cause for your problem. You can test this theory several ways
- Get a USB C to USB3 adapter and plug the drive into that making more power available
- plug a powered hub into the laptop and plug the drive into that (I prefer this test).

I suspect this because your returned drive tested OK. It's only when attached to your laptop you have issues.

The first Seagate ST1000LX015 tested by them had problems so the gave me another Seagate ST1000LX015, when I was returning the second one, it worked with their PC so they gave me my money back
I dont think it is a problem with power since when I connected the drives directly to SATA on the motherboard (my desktop PC with a 500W power supply) it had the same problems.
Can low power cause damage or short term disk slowness? that could explain why it didn't work on my desktop after messing it up on my laptop

Also I tried the external drive (adata) on an USB 2.0 hub without an external power supply and it worked ok (after few hours of idle) at ~38 MB/s while my old failing drive was at ~35 MB/s

Should any external drive go so hot, the new adata is rated at 60°C/50°C (SMART/official site) while on transfer it got to 46
my old failing drive is rated at 55°C and on transfer it is about 30°C

I will test transferring the files through the USB 2.0 hub to see if it will fail too and report back here

 

[Mark RM]

hard drives barely care about heat at all, the numbers you are reporting are fine.

But yes, improperly provisioned power can cause problems on these devices that never seem to go away. I've experienced this a NUMBER of times and this is why I brought it up as a possible issue here.

 

[mirkog]

hard drives barely care about heat at all, the numbers you are reporting are fine.

But yes, improperly provisioned power can cause problems on these devices that never seem to go away. I've experienced this a NUMBER of times and this is why I brought it up as a possible issue here.

Thanks for helping me out.

I've tested it using the USB 2.0 hub and the same issues occurred so I will try to find a externally powered USB 3.0 hub like you recommended.
Could using a Y cable work instead of buying a new hub?

I dont get it why the new drives need so much more power than the old ones (my is from 2012)
Also my laptop should be able to provide sufficient amounts of power to the USB 3.0 ports since it was very expensive, am I right?

 

[Mark RM]

They use more power starting up, this is normally handled with sufficient capacitors on the USB3 to SATA controller board. I actually suspect your Thinkpad is under performing on the USB 3 ports regarding amperage, which is why I didn't suggest a Y cable in this particular case. Normally we use Y cables to draw power from two USB 2 ports (500ma each) to get an amp.

USB 3 usually gives 900ma, I don't think we're getting it and I'm not sure a Y cable would prove or disprove my theory, so I took other roads with my advice to see if we're in the ballpark.

 

[mirkog]

I can confirm this is a Linux issue.
I connected the drive and run an extended test (on ubuntu), when it was done the speed benchark was somewhat ok (~80 MB/s)
Then I went into Windows 10 and run the same transfer again and it finished successfully with normal speed (except on small files of course but access was never blocked)

Then I went to ubuntu and done the transfer to another folder (but same data), after a few minutes iotop shows actual disk write of 0 B/s which sometimes goes above 1MB/s, also the reads stopped at ~1.6 GB which matches my cache configuration of 10%(16GB of RAM total)

I'm using the latest kernel on my installed version mentioned above:
uname -a
Linux mirko-ThinkPad-T470 4.14.6-041406-generic #201712140930 SMP Thu Dec 14 09:31:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Also tried the original Ubuntu MATE 16.04.02 kernel through a live USB

I also noticed that when I format the partition as EXT4, when I mount the partition iotop shows constant ~4 MB/s write speed (I'm not not writing anything to the partition), when I umount it, then it goes back to ~0 MB/s which is very weird and in my knowledge should not happen, I tried this multiple times just to make sure.

My theory is that transferring in linux causes an issue/error which in turn starts an automatic SMART test which uses up all bandwidth and that's why the drive is clicking with no PC -> HDD data transfers

dmesg does not show any error or message, can I somehow debug what is going on?
Maybe you have some specific kernel version or distro in mind to test?
Tomorrow I will test some of the old and very old distros to see if I get the same result

 

[mirkog]

I solved the problem and cant believe how simple it was... Just needed to disable USB autosuspend using the powertop command, Now it transfers data in Linux without any problems too, will test it tomorrow more thoroughly by transferring big amounts of data just to make sure
Theres still the issue of when the EXT4 filesystem is mounted, it writes about 4 MB of data every second (no big deal thought since I wont be using it here), will try to reasearch it and post the answer here if I find one.

Thank you for your help Mark RM.

 

[Mark RM]

I solved the problem and cant believe how simple it was... Just needed to disable USB autosuspend using the powertop command, Now it transfers data in Linux without any problems too, will test it tomorrow more thoroughly by transferring big amounts of data just to make sure
Theres still the issue of when the EXT4 filesystem is mounted, it writes about 4 MB of data every second (no big deal thought since I wont be using it here), will try to reasearch it and post the answer here if I find one.

Thank you for your help Mark RM.

Just happy you got it sort, good work.

 

[mirkog]

It seems like after all it was not an autosuspend issue since the next day I encountered the same issue, today I bought an USB 3.0 Hub with a 2A external power supply but after a few GB I get the same problem of exactly 1 MiB/s write speed (viewed by "iostat -xhdm 2 /dev/sdb " instead of iotop because iotop seems like it is not accurate...)
I was transferring an 4.7 GB file to a newly formatted NTFS partition

However it was working great using the "sudo dd if=/dev/zero of=/dev/sdb bs=10M status=progress" command with an average speed of 130 MB/s (also the above iostat command shows the same)

I'll try to re-partition the disk after zeroing it to see if that will fix the problem (maybe a consequence of initial low power usage which I doubt since the same 4.7 GB file transfer on Windows 10 was working at the same speed of 130 MB/s with 1-2 60 MB/s dips)

UPDATE!
After zeroing it out it worked for a while but again started doing 1 MB/s, here is the iostat log

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0,00  1733,00    0,00  101,00     0,00    11,06   224,20   144,32 1548,67    0,00 1548,67   9,90 100,00
sdb               0,00 19165,00    0,00  141,00     0,00    10,31   149,76   143,09 1291,76    0,00 1291,76   7,09 100,00
sdb               0,00  4078,00    0,00   16,00     0,00    16,00  2047,50   142,84 2319,12    0,00 2319,12  62,50 100,00
sdb               0,00  8156,00    0,00   32,00     0,00    31,99  2047,38   146,95 3813,38    0,00 3813,38  31,25 100,00
sdb               0,00  8035,50    0,00   24,00     0,00    23,99  2046,83   145,41 5396,75    0,00 5396,75  41,67 100,00
sdb               0,00  8156,00    0,00   32,00     0,00    31,98  2047,00   144,62 5235,56    0,00 5235,56  31,25 100,00
sdb               0,00  3187,00    0,00   24,00     0,00    23,99  2047,33   143,84 5218,67    0,00 5218,67  41,67 100,00
sdb               0,00  7900,00    0,00   31,00     0,00    30,52  2016,13   140,51 5235,94    0,00 5235,94  32,26 100,00
sdb               0,00  7520,50    0,00   24,00     0,00    23,99  2046,83   139,42 5276,33    0,00 5276,33  41,67 100,00
sdb               0,00  7381,50    0,00   32,00     0,00    31,98  2047,00   148,57 4896,25    0,00 4896,25  31,25 100,00
sdb               0,00  7903,00    0,00   24,00     0,00    23,99  2047,17   146,95 4936,42    0,00 4936,42  41,67 100,00
sdb               0,00  7903,50    0,00   31,00     0,00    28,96  1912,90   144,07 5171,29    0,00 5171,29  32,26 100,00
sdb               0,00  4079,00    0,00   25,00     0,00    24,99  2047,52   145,76 5268,48    0,00 5268,48  40,00 100,00
sdb               0,00  3951,50    0,00    9,50     0,00     9,50  2047,58   150,05 5390,32    0,00 5390,32 105,26 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   148,26 7742,00    0,00 7742,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   146,39 9868,00    0,00 9868,00 1000,00 100,00
sdb               0,00     0,00    0,00    0,50     0,00     0,50  2048,00   144,49 11460,00    0,00 11460,00 2000,00 100,00
sdb               0,00     0,00    0,00    3,00     0,00     3,00  2046,67   142,78 13976,00    0,00 13976,00 334,00 100,20
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0,00     0,00    0,00    3,50     0,00     3,50  2048,00   132,88 15013,71    0,00 15013,71 285,71 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   129,35 17470,00    0,00 17470,00 1000,00  99,50
sdb               0,00  4079,00    0,00    1,00     0,00     1,00  2048,00   133,32 19612,00    0,00 19612,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2044,00   158,28 21756,00    0,00 21756,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   156,40 22706,00    0,00 22706,00 1000,00 100,00
sdb               0,00     0,00    0,00    0,50     0,00     0,50  2048,00   154,51 24324,00    0,00 24324,00 2000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   152,58 25884,00    0,00 25884,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   150,76 28064,00    0,00 28064,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   148,92 30220,00    0,00 30220,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   147,06 32358,00    0,00 32358,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   145,18 34476,00    0,00 34476,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   143,27 36566,00    0,00 36566,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2044,00   141,34 38634,00    0,00 38634,00 1000,00 100,00
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0,00     0,00    0,00    0,50     0,00     0,50  2048,00   139,47 40240,00    0,00 40240,00 2000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   137,56 41852,00    0,00 41852,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   135,72 44012,00    0,00 44012,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   133,90 46192,00    0,00 46192,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   131,97 48258,00    0,00 48258,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   130,11 50396,00    0,00 50396,00 1000,00 100,00
sdb               0,00  4079,00    0,00    1,00     0,00     1,00  2048,00   131,63 52536,00    0,00 52536,00 1000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2044,00   158,36 53540,00    0,00 53540,00 1000,00 100,00
sdb               0,00     0,00    0,00    0,50     0,00     0,50  2048,00   156,47 55132,00    0,00 55132,00 2000,00 100,00
sdb               0,00     0,00    0,00    1,00     0,00     1,00  2048,00   154,56 56744,00    0,00 56744,00 1000,00 100,00