News WD Fesses Up: Some Red HDDs Use Slow SMR Tech

mattkiss

Reputable
Sep 22, 2016
34
6
4,535
0
I'd be interested to know if their WD Red Pro line also has the same "issue." Looking at a couple of drives for a RAID 1 or RAID 10 array. Drive sizes I'm looking at are 2, 4, 6, and 8 TB.
 

bit_user

Splendid
Ambassador
SMR is a relatively new tactic that HDD vendors use to increase storage density over HDDs that use 'standard' conventional magnetic recording (CMR), but the tech comes with notably slower performance in some workloads than 'normal' hard drives.
I could be wrong, but think CMR stands in contrast to various forms of assisted MR. To that end, SMR is also a CMR technique, as is PMR (perpendicular magnetic recording).

PMR has been dominant for the past 1.5 decades, or so. Prior to that, magnetic polarization was simply within the plane of the disk.

SMR drives are also incredibly slow at random write performance
This is the issue. SMR drives are a good substitute for tapes or "write-once, read-mostly" scenarios, but that's about it.
 
I just bought two of these drives from Amazon. (Wd 6tb red model WWWD60EFAX)

I ran them in unraid as a parity and storage drive. The parity check took about 12 hours. Data speeds ranged from 180MBps to 90MBps with uneven write speeds. Average was 130MBps)

I backed up personal movie/video music and photo collection as well as research documents here. So far zero errors. Im not sure why there are increased errors just because the file system is zfs.

To be honest these drives are designed for NAS likely over a 1 gigabit connection. So the 130MBps is more than fast enough.
 

spongiemaster

Upstanding
Dec 12, 2019
458
181
360
0
I just bought two of these drives from Amazon. (Wd 6tb red model WWWD60EFAX)

I ran them in unraid as a parity and storage drive. The parity check took about 12 hours. Data speeds ranged from 180MBps to 90MBps with uneven write speeds. Average was 130MBps)

I backed up personal movie/video music and photo collection as well as research documents here. So far zero errors. Im not sure why there are increased errors just because the file system is zfs.

To be honest these drives are designed for NAS likely over a 1 gigabit connection. So the 130MBps is more than fast enough.
SMR doesn't impact read speads. It only hurts write speeds. For home file servers and the like where the majority of the work loads are reads, SMR is perfectly fine. Especially for an Unraid server where the parity calculation destroys write speeds anyway, unless you have reconstruct write enabled which comes with its own drawbacks. You really don't want to be running Unraid on a server you are constantly writing large chunks of data to.

If performance is your priority, you should be using the 7200RPM Red Pro drives.
 
Reactions: digitalgriffin

wr3zzz

Distinguished
Dec 31, 2007
16
5
18,515
0
I don't keep up with HDD techs much anymore and was recently shopping for NAS HDD and wondering why Seagate drive listings has so many more letters in addition to the model number, but not WD. Now I get it.
 

bit_user

Splendid
Ambassador
SMR uses perpendicular magnetic recording, making it a form of PMR.
Thanks for the clarification. I suppose I read too much into the "shingled" analogy.

So, maybe you can answer my next question: is the policy to rewrite everything after a modification, until the end of the chunk? I had previously understood that you had to rewrite the entire chunk, but since each track overlaps the previous one, if you can partially overlap it on one write pass, it's not clear to me why you wouldn't be able to partially overlap it on another.
 

bit_user

Splendid
Ambassador
To be honest these drives are designed for NAS likely over a 1 gigabit connection. So the 130MBps is more than fast enough.
That's the sequential write speed. Sequential writes are not the issue with SMR - it's random writes, especially small ones.

Basically, whenever you write even 1 byte in a given ~100 MB chunk*, you have read and re-write the whole thing (or, at least everything including & after the block you touched).

So, if you're just using these drives to hold photos, music, and other stuff that you write once and basically don't touch, it'll probably be fine. If you're using them to hold drive images, which is a perfectly sequential operation, then they'd be great. But, if you're using them for data involving lots of small writes and frequent modifications and deletions, performance is going to be in the toilet.

* Note: I haven't found good info on exactly how big the chunks are. It's drive-specific and I think manufacturers are not forthcoming about it. However, the info I've seen suggests chunk sizes on the order of 100 MB.
 

bit_user

Splendid
Ambassador
If performance is your priority, you should be using the 7200RPM Red Pro drives.
I've also seen strong endorsements of Seagate Iron Wolf Pro drives.

However, my latest drive is a WD-branded HGST Gold drive, and it's great. Much better than the non-HGST WD Gold drive it replaced. I wish I could replace all of my old drives with ones like it, but I'd rather not spend the $$$. Annoyingly, a RAID is only as fast as its slowest drive.
 

PaulAlcorn

Senior Editor
Editor
Feb 24, 2015
763
125
11,160
0
Thanks for the clarification. I suppose I read too much into the "shingled" analogy.

So, maybe you can answer my next question: is the policy to rewrite everything after a modification, until the end of the chunk? I had previously understood that you had to rewrite the entire chunk, but since each track overlaps the previous one, if you can partially overlap it on one write pass, it's not clear to me why you wouldn't be able to partially overlap it on another.
I once had an SMR explainer written on another site that has since closed (I miss Tom's IT Pro). The article was probably too long then, and is probably still too long now. In either case, here's a relevant bit -

The process of writing data to an SMR HDD is straightforward at first. The drive writes sequential data normally, and the tracks are overlapped (shingled) during the initial write process. However, if new (or modified) data needs to be placed near existing data, any modification of the existing sectors will overwrite the neighboring shingled tracks. Thus, an overwrite triggers a read-modify-write procedure due to the inevitable impact on the adjacent overlapped tracks. Unfortunately, random write activity also triggers this process, because any data dropped into a sector at random will affect overlapping tracks.

This overwrite process begins with the HDD reading all data that will be overwritten, then the data in the following sectors/tracks that are affected, and moving it to a cache. In the case of Seagate SMR drives, the cache consists of a section of the platter (rumored to be 20GB) used as persistent storage. The drive uses the persistent cache location on the platters to hold data as it reorganizes it. Once the data is reorganized, the drive rewrites it sequentially back to the home location.

An SMR HDD also utilizes DRAM caching to aggregate and sequentialize incoming random write data before writing it to the platters. The drive also uses the cache section of the platter to provide persistence for the data held in the volatile DRAM cache. The limited size of the platter-based cache speeds data operations because it reduces head movement as it transfers data into, and out of, the cache. The persistent cache can also theoretically be NAND flash-based, and it wouldn't be surprising to see that technique come to market to help address the inherent performance limitations associated with SMR.

The drive rewrites modified sectors and tracks when it returns the data from the cache to the home location. This process would require an entire re-write of the HDD (from the affected track) for every re-write operation if the drive were shingled end-to-end. The drive has tracks that are grouped into bands to avoid this scenario, and any track modification only requires reorganization to the end of the band.

HGST is utilizing 256MB bands in its inaugural offerings. Seagate indicates that the sizes of the bands are adjustable for custom drive workloads and applications. These adjustments will likely result in drives developed specifically for targeted use-cases as the technology progresses.


SMR's overlapped track structure is optimized for large sequential read/write operations, but the technique will struggle with any type of random write data.

To accomplish the complex data placement tasks, SMR HDDs remap the LBA tables. The graphic above is an example of the modified LBA mapping techniques utilized on an SMR drive. Even if the host writes data to the drive sequentially, the drive may not initially write the data in a contiguous fashion due to the LBA table remapping. This means that some portions of files that would normally be concentrated on the outer edge of the platter, for instance, could possibly end up on other portions of the platter as well, which will can affect performance consistency. This brings forth new challenges and an additional layer of complexity (in comparison to SSDs) that must be overcome to keep performance within expectations.

The read-write-modify process is very similar to how SSDs utilize a flash translation layer in tandem with overprovisioning, and it incurs a heavy performance penalty in some scenarios. Many of the engineering techniques utilized to abstract the LBA space, such as using dynamic LBA mapping in favor of static mapping, come from established SSD technology. It is ideal if this process of sequentializing incoming random data, and the read-write-modify process, happens at the software layer (as explained on the following page).
 
Last edited:

PaulAlcorn

Senior Editor
Editor
Feb 24, 2015
763
125
11,160
0
SMR doesn't impact read speads. It only hurts write speeds. For home file servers and the like where the majority of the work loads are reads, SMR is perfectly fine. Especially for an Unraid server where the parity calculation destroys write speeds anyway, unless you have reconstruct write enabled which comes with its own drawbacks. You really don't want to be running Unraid on a server you are constantly writing large chunks of data to.

If performance is your priority, you should be using the 7200RPM Red Pro drives.
The problem stems from the long periods of time that the drive requires for background operations that restore performance. Just like an SSD, they have a garbage collection-like process, but it happens at HDD speeds. As in utterly terrible speeds. This causes the drive to pause before fulfilling requests, and that apparently happens for such a long period of time that the drive gets kicked out of the array (marked as unresponsive/failed). There was a similar problem with Green drives back in the day.

This also answers the previous reader's question about why this impacts ZFS.
 
Reactions: bit_user

spongiemaster

Upstanding
Dec 12, 2019
458
181
360
0
The problem stems from the long periods of time that the drive requires for background operations that restore performance. Just like an SSD, they have a garbage collection-like process, but it happens at HDD speeds. As in utterly terrible speeds. This causes the drive to pause before fulfilling requests, and that apparently happens for such a long period of time that the drive gets kicked out of the array (marked as unresponsive/failed). There was a similar problem with Green drives back in the day.

This also answers the previous reader's question about why this impacts ZFS.
A quick google of the problem seems to be that users are complaining that these drives error out when trying to perform an array rebuild. And even before the errors, the process is unbearably slow. This would make perfect sense based on how write intensive a rebuild is. Seems bizarre that WD didn't catch these problems in testing. The drive literally say NAS on the label and RAID is all over the data sheets.

For a home server that isn't likely to be getting hit 24-7, I don't think the garbage collection should be a big deal. How often are you going to be overwriting data on a home server? You dump your photos and media files on it, and then they just sit there.

If you're using standard Red drives with only a 3 year warranty, and 8 drive limit to an enclosure for your business, the question has to be why? Is your data not worth the few extra bucks per drive for the Pro versions?
 

King_V

Distinguished
Well, fantastic. Just finally dipping my toes into setting up my NAS, and while I haven't unpacked them yet, I did order and receive these:
WD Red WD40EFAX 4TB 5400 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
Though at the time, two weeks ago, they were $89.99.

The NAS is only a two bay unit, and it's not going to be anything like professional use, but damn, I don't want NAS drives that are potentially troublesome when used in RAID for NAS. You know, which is what they're marketed and supposedly designed for.

I hope a way to identify these particular drives shows up soon.
 

stoatwblr

Distinguished
Sep 12, 2011
32
7
18,535
0
Most of what happened over the weekend is due to my work and research over the last 3-4 weeks, collating both my own experience and dozens of other reports into something coherent for market regulators and journalists to understand.


I really suggest that if people want to get a good understanding of what happens inside a DM-SMR drive, that they look at the HGST presentation to the OpenZFS meeting in paris, 2015. It's 32 minutes and very understandable.

View: https://www.youtube.com/watch?v=a2lnMxMUxyc&list=UU0IK6Y4Go2KtRueHDiQcxow&index=107


You'll note that the engineer (Manfred Berger) from HGST stated during this presentation that even with CMR staging zones they found the performance of DM-DMR was so wildly variable and would get so bad, that they declined to bring these drives to market, concentrating only on Host Aware and Host Managed drives (HA-DMR and HM-DMR)

In other words: WD and SG knew there are major problems with DM-SMR even before it hit the market, and shipped the things ANYWAY.

Even worse, they've gone out of their way to conceal the fact that drives are SMR and flat out stated to complainants that they WILL NOT DISCLOSE whether a drive is SMR or CMR internally.

The level of cynicism that's involvd here is beyond belief. How they thought they'd get away with this, I'm not really sure.... (and honestly for the last 2 years they HAVE gotten away with this, because people thought they had individual bad units)



This is legally actionable behaviour by marketers (misrepresentation of product) - and the fact that both Seagate and WD have been essentially doing the same thing and giving almost the same answers to complainants indicates cartel behaviour.



On top of all this: There's a major firmware BUG in WDx0EFAX drives which causes them to throw IDNF errors during ZFS resilvering (Sector ID Not found according to smartctl -x log views) that in turn causes ZFS to throw them out of the array. WD deny that the bug exists.

How do I know it's a bug and not a bad drive?
1 drive might be a bad drive.
2 is rather less likely to be one.
3 drives, all throwing identical internal errors at more or less the same spots is highly UNlikely to be bad hardware
 

stoatwblr

Distinguished
Sep 12, 2011
32
7
18,535
0
If you are not sure about your drives, a few rules of thumb can help:

- Anything with large cache (256MB and up) is highly likely to be SMR but you need to benchmark to be sure. DM-SMR requires large amounts of cache.

- Anything that has changed model number suffix (EG: EFRX to EFAX for WD-REDs or even somehting as insinuous as ST3000DM006 to ST3000DM007 for Seagate) and acquired large cache, BUT ALSO lost a chunk of weight is extremely likely to be DM-SMR - but you'll need to benchmark to be sure. The reason for the weight loss is that SMR allows removal of one or more platters for the same drive capacity.

- Any rotational drive that reports TRIM functions IS DM-SMR (EG: WDx0EFAX)

- However BEWARE: not all DM-SMR drives report TRIM functions (EG: ST3000DM007)


Check the "platter database" - https://rml527.blogspot.com/

This is about the only resource available at the moment, because WD and SG are still refusing to provide tables of which drives are DM-SMR and which are not.

I wouldn't be at all surprised if they attempt to shut it down...
 
The problem stems from the long periods of time that the drive requires for background operations that restore performance. Just like an SSD, they have a garbage collection-like process, but it happens at HDD speeds. As in utterly terrible speeds. This causes the drive to pause before fulfilling requests, and that apparently happens for such a long period of time that the drive gets kicked out of the array (marked as unresponsive/failed). There was a similar problem with Green drives back in the day.

This also answers the previous reader's question about why this impacts ZFS.
That leaves me with the difficult decision of returning the drives.

Unraid uses xfs, but rebuilding it sounds like it could be an issue. I have one as a parity drive and one as a replacement data drive.

My other option is buy a seagate iron wolf as a second parity, but I heard horrible reliability stories about the rosewood design with head failure.

The pro drives aren't an option as they are nearly twice the price per gigabyte.

Hgst maybe?
 

2Be_or_Not2Be

Distinguished
Aug 23, 2013
866
16
19,365
115
To me, SMR drives usually aren't worth the small price savings. However, WD actually hiding the fact they are using SMR drives in their NAS drive line-ups & refusing to identify the recording method is a big negative against them. Back in the day, you knew what you were getting in the WD Black series. Now, it's enough to make you question all of their line-ups.

Large, cost-efficient SSDs can't get here soon enough! (Of course, then I'll be equally disappointed to find out even the TLC SSDs are being swapped out for QLC - flash's SMR-equivalent)
 
Reactions: King_V

Uniblab

Distinguished
Jul 1, 2009
29
0
18,530
0
WD admits that it sells hard drives with inferior SMR technology without disclosure in advertising or specification sheets.

WD Fesses Up: Some Red HDDs Use Slow SMR Tech : Read more
Wish you would have taken just a lil more time to explain how shingled HD's work. I still cant get past how its possible for a platter to have separate read and write "Tracks". Arent there tracks that a read and write head, writes and reads to/from? How can a magnetic track that was never written to now contain information that can be read from. There is something new going on, or something is not explained correctly.
 

bit_user

Splendid
Ambassador
HGST doesn't exist anymore. It's WD.
Well, not quite.

Until very recently, WD kept their legacy and HGST design groups separate, for regulatory reasons. Even now, you still find some WD-branded drives coming from one group or another. I just had an experience where I replaced a WD Gold drive from 2017 with a much better one from 2019, and it turned out the 2019 drive was a HGST, while the 2017 model was not.

And when I say better, I mean the newer model had better transfer speed (in spite of being the same capacity), was quieter, and had lower power consumption & temperatures. I can't quite compare the raw error rates, because they supported different sets of statistics. But, the error rates on the 2017 drives were worryingly high, and one of the four failed in the first 100 hours of use (hence, why I replaced it).

Hgst maybe?
The drive I can vouch for is the WD4003FRYZ. Yeah, it's not cheap. I wish all 4 of mine were that model, instead of just the one replacement.
 
Last edited:

bit_user

Splendid
Ambassador
Thanks for investigating & posting.

On top of all this: There's a major firmware BUG ...

How do I know it's a bug and not a bad drive?
1 drive might be a bad drive.
2 is rather less likely to be one.
3 drives, all throwing identical internal errors at more or less the same spots is highly UNlikely to be bad hardware
Something like that could also be due to a hardware design flaw or systematic manufacturing defect.
 

bit_user

Splendid
Ambassador
Thanks for posting.

HGST is utilizing 256MB bands in its inaugural offerings. Seagate indicates that the sizes of the bands are adjustable for custom drive workloads and applications. These adjustments will likely result in drives developed specifically for targeted use-cases as the technology progresses.
Ah, good to know. The writeup I saw that talked about 40 - 100 MB bands was very old.

Anyway, it also seems to confirm my understanding that it's only the tracks after the modified block that need to get rewritten - not the entire band (unless the modified block is at the beginning). That makes sense.
 
Last edited:

ASK THE COMMUNITY

TRENDING THREADS