News Unpowered SSD endurance investigation finds severe data loss and performance issues, reminds us of the importance of refreshing backups

[bit_user]

(Opinion) My expectation is for data to degrade after a year, corruption to begin after a year. I don’t trust TLC, I don’t trust QLC for archiving. I can no longer buy MLC, I would if I could.

Hmmm... it does seem that even datacenter drives are either SLC or TLC. I could've missed them, but I'm not seeing MLC models, even within the past 5 years or so.

 

[bit_user]

The safest way seems to be having multiple copies of your dataset, at different physical location, having redundancy at each location, and keeping the data online on a filesystem with data checksumming and periodic scrubbing (i.e. ZFS). But that's tedious and expensive.

I use mdraid, on Linux. My distro has a monthly cron job that scrubs any mdraid volumes you have.

BTRFS has checksums at the filesystem level, but checking those requires taking the filesystem offline. I think they're checked each time you read a file, so you could theoretically run an online job that just walks the directory tree and reads all of the files.

I recently plugged in a cold-storage harddrive after a couple of years offline, and it suddenly had unreadable sectors, even though it's been stored in a shock-absorbant case in relatively controlled climate. Obviously just a single datapoint which shouldn't be extrapolated - but it did make me reconsider my archival strategy.

I had a 5-disk RAID of 1 TB WD Black drives. I waited too long to replace them (a whole 10 years!), but they never showed any SMART errors and mdraid never found any inconsistencies or RAID parity errors.

These were the same drives I mentioned before, where I had knocked one off a table onto a hard wood floor, almost 3.5 feet below. It completed a full, extended self-test, just like nothing had ever happened. Still no errors visible in the SMART logs.

 

[hurnii]

Kinda odd not to include a link or tell us the brand & model number.

Am not going to advertise for a drive that's apparently no longer available. (If I could still buy a couple more, I'd say who they are.)

I had wondered about this and done some poking around, but what I found seemed to indicate that Optane's data retention was even worse than NAND!

Re: Optane: I had no idea! Looked around and found:

Optane is only rated for a 3-month data retention

Ouch! Guess I'd better power up that P5000X drive I have lying around, and see how bad it is. Does 3 months even count as "Static" memory? Sounds more like DRAM, with a 3 month long refresh cycle. No wonder Intel isn't making them, anymore (well, among other things).

bit_user:

(quoting me)
Can tell from the overkill of putting this drive into a retired laptop "just in case" - I take my "Oh, Honey..."'s seriously. ("How can we, prevent this, from happening, again?" - mantra from my time in the US Navy)
(end quote)
Backup to hard disk or cloud. That's your best bet. Anything else is a crapshoot. At least, if you're going to leave it powered-off.

From my wife's point of view, "this" (the thing to be prevented) was the inability of her Laptop to boot. Short of going back to an HDD, I did what I could. Powering it up, regularly (maybe once a year?) running trim, leaving it up for say, a couple of days (enough time for "StaticDataRefresh technology to do it's thing), and then, running chkdsk /r, leave it up for another day, and shut it back down until the next year should help. Since already have an image of the drive, backed up in multiple places, maybe, every 5 years, do a low level format of the drive (preferably using an SSD tool) to get all of the usable cells back to 0 state), write the image back to it, and shut it back down?

Am not sure whether anyone ever wrote a guide on how to get the maximum powered down lifespan out of an SSD, probably not. Do know, keeping the powered down SSD cool helps it retain the charge states.

 

[bit_user]

Re: Optane: I had no idea! Looked around and found:
...
Ouch! Guess I'd better power up that P5000X drive I have lying around, and see how bad it is. Does 3 months even count as "Static" memory? Sounds more like DRAM, with a 3 month long refresh cycle. No wonder Intel isn't making them, anymore (well, among other things).

Somehow, I think it just can't be that bad. If it were, it'd be "common knowledge", because Intel did sell enough Optane drives to consumers that lots of people would've been caught out by it. I think the 90-day thing is probably a lot more like either a minimum they felt like they had to say or maybe an extreme corner case. Do please let us know what you find.

BTW, is it a P5800X? I have one of those. Only 400 GB, which is plenty for a boot drive. I bought it when they announced the product line discontinuation, but then it sat on the shelf while I tried to decide what to do with it. I've finally reached the point where I'm ready to install it in a machine I use daily.

Powering it up, regularly (maybe once a year?) running trim, leaving it up for say, a couple of days (enough time for "StaticDataRefresh technology to do it's thing), and then, running chkdsk /r, leave it up for another day, and shut it back down until the next year should help.

Yeah, that uncertainty is what I hate, because I have a couple machines I rarely use. In the past, I figured just reading all the blocks might be enough to trigger the drive to copy or refresh all the weak ones.

Maybe I will switch to using the Linux badblocks tool, which has a mode where it reads the old block, writes a test pattern, reads back the test pattern, and then puts your original data back. That's a couple more writes than I'd like, but it does guarantee the block is refreshed!

Also, I just have to point out that certain people in this thread have been implying the only thing to worry about is a corrupt file. Of course, that's not true. Some of what the drive holds is filesystem datastructures. If those get corrupted, you could lose vast numbers of your files. Statistically, it's a lot more likely that isolated errors land in regular file data, but it's really just a roll of the dice away from hitting the filesystem.

every 5 years, do a low level format of the drive (preferably using an SSD tool) to get all of the usable cells back to 0 state), write the image back to it, and shut it back down?

Like I've been discussing with @jp7189 , reformatting is probably a waste of time. If you want to force the drive to be refreshed, either use a tool which does a block-level refresh or just copy the filesystem image off and then back onto the drive.

If you have any way to force it to run TRIM, after that refresh, that would be a really good idea, because the underlying FTL (Flash Translation Layer) would then think that your volume is 100% full, even if its not. TRIM tells the drive which blocks you're not using, so the FTL can add them to its pool of reserve blocks to use for wear-leveling.

Am not sure whether anyone ever wrote a guide on how to get the maximum powered down lifespan out of an SSD, probably not. Do know, keeping the powered down SSD cool helps it retain the charge states.

Yeah, it's the kind of thing where most of us are flying blind. You don't really know what's going on until errors start popping up. I'm sure people who work on these drives have more insight. There ought to be ways they can see low-level details, like rates of charge decay. I think modern NAND must do some sort of calibration involving cells that hold a known value and then compensating the rest of the cells in the block by their voltage. The SSD's controller probably has ways of seeing this information and it'd give much deeper insight and earlier feedback into the state of the drive.

Interesting discussion, for sure. Thanks for sharing your details!
: )

 

[snemarch]

I use mdraid, on Linux. My distro has a monthly cron job that scrubs any mdraid volumes you have.

BTRFS has checksums at the filesystem level, but checking those requires taking the filesystem offline. I think they're checked each time you read a file, so you could theoretically run an online job that just walks the directory tree and reads all of the files.

I have a flash-based Linux fileserver, which used to run a couple of (Samsung QVO, not really optimal) SSDs in mdraid, until I had a little encryption oopsie – that was using mdraid and xfs. Now it's temporarily serving two (non-mirrored) NVMe drives, still with xfs. I'm building a new system later this year, when I've saved up a bit of money and figured out which parts run at reasonably low power consumption... and then it'll be TrueNAS and zfs.

I also have a Synology 1522+ with spinning rust storage, which is used as a backup destination. That one's running btrfs, and a montly scheduled data scrub. I'm not sure how I feel about btrfs in general, there's a fair amount of stories about it being buggy... but Synology doesn't use its raid features, and instead relies on a mix of mdraid and lvm, so... 🤷‍♂️

I had a 5-disk RAID of 1 TB WD Black drives. I waited too long to replace them (a whole 10 years!), but they never showed any SMART errors and mdraid never found any inconsistencies or RAID parity errors.

But those were kept online, right? My failing drive (a 640gig WD Caviar Black) had been offline. i do wonder what the failure mode is, the case it was in has been sitting in a closet, so it hasn't been bumped around or subjected to harsh temperatures, humidity or anything.

 

[bit_user]

I'm not sure how I feel about btrfs in general, there's a fair amount of stories about it being buggy... but Synology doesn't use its raid features, and instead relies on a mix of mdraid and lvm, so... 🤷‍♂️

As long as you stay away from the more exotic features and don't use bleeding-edge kernels, I think BTRFS is quite safe. Some of the stories out there are people blaming BTRFS, when actually their hardware is at fault. Because of its use of checksums, it can detect errors that other filesystems would've allowed to silently accumulate.

ZFS users tend to be more sophisticated, simply because it's a self-selecting set of people who are typically deploying it on an array. However, several distros have used BTRFS as the default root filesystem, meaning you have lots of cases where less knowledgeable users stumble over BTRFS error messages. They also aren't all using the best hardware. Some might even have DRAM errors that are triggering BTRFS checksum mismatches.

But those were kept online, right? My failing drive (a 640gig WD Caviar Black) had been offline.

I left them online, for several years. Then, started turning off the machine, when not needed.

 

[snemarch]

As long as you stay away from the more exotic features and don't use bleeding-edge kernels, I think BTRFS is quite safe. Some of the stories out there are people blaming BTRFS, when actually their hardware is at fault. Because of its use of checksums, it can detect errors that other filesystems would've allowed to silently accumulate.

I haven't followed it closely, but I had the impression that even relatively standard stuff like RAID has had some pretty nasty bugs. I would probably still choose btrfs if I were going to run standard Linux on my server rebuild, though – snapshots are such a handy feature. But given that my few non-storage workloads run fine in containers, it almost seems like a no-brainer to go with TrueNAS and get ZFS. I probably wouldn't muck around with ZFS on a standard Linux system, though...

ZFS users tend to be more sophisticated, simply because it's a self-selecting set of people who are typically deploying it on an array. However, several distros have used BTRFS as the default root filesystem, meaning you have lots of cases where less knowledgeable users stumble over BTRFS error messages. They also aren't all using the best hardware. Some might even have DRAM errors that are triggering BTRFS checksum mismatches.

Yeah, while btrfs probably has had it's share of nasty bugs, it's not to blame for bad hardware. Or hardware that outright lies about e.g. flush completion 🫠

 

[bit_user]

I haven't followed it closely, but I had the impression that even relatively standard stuff like RAID has had some pretty nasty bugs.

It might sound standard to you, but I consider that one of its more exotic features. Its built-in RAID functionality has been labeled experimental, for as long as I can remember.

I use a non-RAID BTRFS volume atop mdraid. Works fine. Yeah, you miss out on a few features of having RAID integrated into the filesystem, but it's nothing I miss too much.

snapshots are such a handy feature.

I love snapshots for doing backups. First, they give you a way you can have light-weight pseudo-backups within your volume. This can protect against accidental file deletion, and the like.

Secondly, if you backup from a snapshot, you're copying a static image of the filesystem that was created atomically - not backing up a filesystem that's currently in flux. The difference is somewhat academic, but it can potentially avoid your backup capturing inconsistent or invalid state of some application data.

Yeah, while btrfs probably has had it's share of nasty bugs, it's not to blame for bad hardware. Or hardware that outright lies about e.g. flush completion 🫠

It's been around for a long time, over which there have been some nasty bugs. If you don't run a bleeding edge kernel, you're not likely to encounter any of them.

 

[hurnii]

BTW, is it a P5800X? I have one of those. Only 400 GB, which is plenty for a boot drive. I bought it when they announced the product line discontinuation, but then it sat on the shelf while I tried to decide what to do with it. I've finally reached the point where I'm ready to install it in a machine I use daily.

Yes, it is. Is actually a Dell "branded" P5800X SSDPF21Q400GB01 with U.2 interface.
Got it soon after the discontinuation notice, and, occasionally check to see how much they're going for, now. (Price keeps going up, faster than inflation.)
Bought a separate PCIe v4 x4 card to mount it on, and it's been waiting for me to tear down a current build with an older 905P (PCIe v3x4) so that it can replace the older drive. The 5800X is less than half the size of the 905P, but it's roughly 4x as fast.

 

[bit_user]

Yes, it is. Is actually a Dell "branded" P5800X SSDPF21Q400GB01 with U.2 interface.

Yup. As far as I can tell, they never sold these as half-height, half-length PCIe cards. Only U.2 form factor, which is actually fine with me, because it's the front of my case where I have direct airflow!

I actually got a M.2 adapter/cable for mine, even though my motherboard has an oculink connector. The issue with that is it's connected through the chipset, while one of the M.2 slots is CPU-direct. Doesn't make a big difference, but you can actually see the impact on QD1 IOPS. Since that's its party trick, I figured "why not let it do what it does best?"

occasionally check to see how much they're going for, now. (Price keeps going up, faster than inflation.)

Oh, well... in my case, I bought it from an authorized reseller, which means I paid about $1100ish. I did a search on ebay, just now, and I see that "new" ones, even shipping from the US, are currently going for less than half as much (the two best are $445 and $480, free shipping). Doh!

That's one reason I decided to go ahead and use it. I got lucky with a Radeon VII, that I sold for a profit, back in the pandemic/crypto boom years. I thought this might turn out to be another rare tech item that increases in value, but sadly not. Now, I'm not sure it ever will. By the time inventories are exhausted, tech will have moved on and CXL-based drives will probably be the thing in demand.

Bought a separate PCIe v4 x4 card to mount it on, and it's been waiting for me to tear down a current build with an older 905P (PCIe v3x4) so that it can replace the older drive.

It does like active cooling. It can use up to 21W, while the operating temperature range is only up to 70 C. That's why I'm happy to mount mine in the drive cage, right behind a 140 mm intake fan.

https://www.intel.com/content/www/u...b-2-5in-pcie-x4-3d-xpoint/specifications.html
​