Question Could Allocation Unit Size impact data integrity on an archive HDD ?

[MaxT2]

Context
Considering a large (rotating) hard drive. This drive will hold large *.rar archives files. (The files already exist and will be transferred from some of my older drives.)

*.rar files come with a "recovery record" (RAR equivalent of .par files) of usually 3% (sometimes I set 5%) which would allow them to get repaired in case they are corrupted. But if the damage on a given file is too large, it wouldn't be able to repair. (This always depends on the size allocated to recovery data).

From my understanding, a 2 MB Allocation Unit Size means that the system will read/write 2 MB at once on this drive.
From past tests, I think that this do speed up mass testing of the archives (which can take more than 24h). (Though I haven't done enough tests with all other parameters being equal to be 100% on this.)
So, on this drive, 2 MB would make sense.

Question
But the question I wondering about right now is:
If some data fails anywhere on the drive, does a 2 MB Allocation Unit Size also mean that 2 MB would be corrupted at once? (In any case or even in some cases?)
Or is the drive still able to read most of the 2 MB except for let's say a few corrupted bytes?

<non essential content removed by Moderator>

 

[Ralston18]

Start by providing what you believe to be the answers to your questions.

Show the calculations, cite references, add explanations, etc..

For example "From past tests...." What tests?

Your question(s) seem to be either a homework question or research project.

Per Forum rules we do neither.

 

[MaxT2]

Start by providing what you believe to be the answers to your questions.

Maybe I explained context too much: The only actual question in my post could be summarize this, I would like to understand the precise behaviour of drives in regards to allocation: Considering 2 MB Allocation Unit Size, if, within a given 2 MB "unit" (assuming it's a very random one in middle of the file, so it's fully used), a few bytes (let's say 5 bytes) are corrupted/unreadable, does this mean that the reading of the rest of the related 2 MB would "fail"? So I should consider that 2MB of a given file would fail at once? Or would the drive still read the 2 MB and only those 5 bytes would count as corrupted? Then, maybe there era different types of failure?

I'm not expecting anyone to do performance tests, but I expect that people who know how drives work can reply this question briefly.

(By way systems the may use these drivers are Windows 10 Pro and Windows 11 Pro and partition system is NTFS.)

The mention of past tests is only part of context and part of the reason why I go with 2 MB allocation on some archive disks. Tests made during my normal usage of archive drives (which means running batch-tests on the archive files and the full test taking something around 26 hours on a drive or more like 20 hours on a similar drive ). I did not make any prepared tests taking notes or such. And there's a chance that performance benefits are lesser than my impression or are due to other factors. But isn't it what the theory says? That larger allocation may improve performances on heavy read/write operations on large files as more data are read at once/in a row?
(I have now edited first post to make the separation between context and question clearer.)

 

[wyliec2]

IIUC your fundamental question is does a bad spot on a hard drive result in the entire cluster (defined by allocation size) being unreadable/corrupt?

I don't know the answer, but it's and interesting question since I have several large HDDs specifically for storing large (>10 - 100 GB) files. Consequently, I have increased the allocation size.

Again, no expert on the physical details of disk (I do understand tracks and cylinders, etc.)

Considering for example a disk defect renders 1000 bytes unreadable.

What is the impact with given file allocation sizes:

4 KB - if 1000 bytes within it are bad, is the whole 4KB cluster unreadable?

2 MB - if 1000 bytes within it are bad, is the whole 2 MB unreadable?

 

[MaxT2]

IIUC your fundamental question is does a bad spot on a hard drive result in the entire cluster (defined by allocation size) being unreadable/corrupt?

Yes, thank you, that is a good summary. I hope someone actually comes with the answer.

 

[USAFRet]

While not an authoritative answer....

I posit it does not make a real difference.
Physical corruption of the platter surface doesn't respect "sector boundaries".

Like if your neighbor has weeds. Left to their own devices, eventually they will cross over into your yard.

As far as the data....5 bytes of corrupted data can render an entire file unusable.
Or maybe just a bad spot in a picture.

 

[MaxT2]

Physical corruption of the platter surface doesn't respect "sector boundaries".
Like if your neighbor has weeds. Left to their own devices, eventually they will cross over into your yard.

Thanks but, this a wrong approach to the answer. Your example means that if there is corruption somewhere, there might be two adjacent sectors corrupt even if they are not fully corrupted.
But that does not reply to the initial question which would rather be "is a sector entirely corrupt/or impossible to read if some (not all) data it contains is corrupt?".
(or, if there's any weed in the yard, is the whole yard considered a disaster?)

(Edit: And this actually makes the numbers worst because if the 5 corrupted bytes are between two 2 MB sectors, would I end up with 4 MB of non-readable data?)

As far as the data....5 bytes of corrupted data can render an entire file unusable.
Or maybe just a bad spot in a picture.

Or some characters in a text... but...
It depends, talking about archive files only, the more recovery data is included, the larger some corruption quantity can be repaired.

 

[USAFRet]

A drive has replacement sectors built in.

When corruption is detected, it (hopefully) remaps that data to one of the previously unused replacement sectors.Leaving the corrupted space alone.

So whether it is a tiny part of that sector, or the whole thing...is sort of irrelevant.
The OS and physical drive will no longer use it.

 

[MaxT2]

A drive has replacement sectors built in.

Maybe, but the issue is that corruption exists and happens, much more often than zero times, also at file level. (Even on an untouched, protected and shelves drive.)

When corruption is detected, it (hopefully) remaps that data to one of the previously unused replacement sectors. Leaving the corrupted space alone.

But would it "block" whole sectors?

Also, I'm not sure at what moment a drive detects corrupted data (other then when you read it or do a chkdsk or so) and it is certainly not able to fixe files. That's the job of the archive software and of the recovery data. So, drive reallocate data is only I'll, let's say, disk-surface-maintenance, it's not magic file-fixing... regarding files integrity, I think it is only moving the problem from a place to another . (I'm excluding any involvement of RAID or stuff like that, as I'm talking about archives, very 'cold' archives, and not system resilience.)

 

[USAFRet]

Maybe, but the issue is that corruption exists and happens, much more often than zero times, also at file level. (Even on an untouched, protected and shelves drive.)

Yes, more than zero.
But obviously, not unlimited.

Eventually, it runs out of replacement sectors, and data actually does get corrupted.
A wise person will have replaced it long before that happens.

But would it "block" whole sectors?

Yes.
A bad sector is a bad sector. Not 'partial'.


A couple of years ago, I had a 16TB Toshiba Enterprise go from 0 to 14k+ 'bad sectors in just a few days.
It was 7 months old.

 

[MaxT2]

Yes, more than zero.
But obviously, not unlimited.

Eventually, it runs out of replacement sectors, and data actually does get corrupted.
A wise person will have replaced it long before that happens.

But that's absolutely not the right approach to it. I'm trying to understand if I can lower the risk to get corrupted data... while hoping to keep performance optimise somehow, (because but a full integrity test (not talking about chkdsk or such but about unrar.exe t ....) of a whole 10 to 14 disk takes 24+ hours already) .

With a goal that would ideally be preservation of 100% files and I think I achieve preservation of 99.9999999% of my archives over more than 20 years.

What you are saying it when enough files are damage, one would replace the disk. In this topic, you should think about the damages to the files first, the disk itself is the element that doesn't matter, because it's just the tool and it is replaceable indeed, (not too often hopefully).

A couple of years ago, I had a 16TB Toshiba Enterprise go from 0 to 14k+ 'bad sectors in just a few days.
It was 7 months old.

I will have nightmares now.

 

[USAFRet]

I will have nightmares now.

This was in my QNAP NAS.

Toshiba warranty replaced the physical drive.
My backup routine replaced the data.

But that's absolutely not the right approach to it. I'm trying to lower the risk to get corrupted data.
With a goal that would ideally be preservation of 100% files and I think I achieve preservation of 99.9999999% of my archives over more than 20 years.

Multiple backup copies, on multiple physical devices.
They won't all go bad at once.

 

[MaxT2]

Multiple backup copies, on multiple physical devices.
They won't all go bad at once.

Yes, of course, but this is moving away from initial question and just reminding the basics.

 

[USAFRet]

Yes, of course, but this is moving away from initial question and just reminding the basics.

A bad sector is a bad sector.
And shall be remapped, and the original bad sector ignored.

 

[evermorex76]

A drive has replacement sectors built in.

When corruption is detected, it (hopefully) remaps that data to one of the previously unused replacement sectors.Leaving the corrupted space alone.

So whether it is a tiny part of that sector, or the whole thing...is sort of irrelevant.
The OS and physical drive will no longer use it.

If the corruption happens and the block can't be read, the data can't be moved. The block will get reallocated (eventually, hopefully) and never used again, but the data is lost unless the drive detects the block is beginning to fail and moves it ahead of time. If it's an archive drive and some bits degrade while it's sitting unpowered, they are more likely to be lost completely as the drive is never checking them. (Even powered, I'm not sure a mechanical drive is going to be checking the blocks automatically.)

This question is not actually about the drive, it's about the OS. The drive can read every block that isn't bad. It doesn't care about allocation units/cluster size. If the OS says to read 2MB worth of blocks, the drive can read all but the bad one and will return the data to the OS, but it's up to the OS to do something with it.

I would, barring a definitive answer, assume that the OS normally requires all blocks of an allocation unit to be readable in order to read that allocation unit as valid. The OS wouldn't know what to do where the data was lost. It might be possible for recovery software (even chkdsk) to read the good blocks and create a recovered file from it (00000001.chk for example), but depending on the data type, that might not do any good. Most of us have probably had chkdsk create a ton of "recovered" files from things like orphaned blocks but they're just parts of files and unusable. Chkdsk and other tools can't reintegrate it into the original file with some zeroes to replace the bad data, giving the recovery record a chance to work. But chkdsk would probably try to create a recovery file containing the entire contents of the original RAR file, minus the bad data. Maybe the recovery record could work with that, but depends on just how big these archive files are. Would you have the space for chkdsk to do that?

So if the RAR recovery record can't deal with at least 4MB of damaged data, that might not be a good size to use (either use a smaller cluster size, or a larger recovery record). (4MB to account for damage that happens to pass from one allocation unit to another, last block of one to first block of another.)

And of course, what happens if the bad blocks affect both some data and the recovery record? It would be like having the parity block in a RAID array get corrupted at the same time as the data. No way to rebuild without the full record.

Maybe the best idea is archiving to two drives (RAID or separately), if it's that important. If you don't care that much or you're going to test these archives pretty regularly though, like every few months, one drive is probably fine. Consider the music industry which recently discovered their hard drives with stuff like master tracks from decades ago had gone bad because they hadn't been testing them or moving the data to newer storage. Even a yearly test would probably be fine if the data isn't life-saving or financially-binding or something, as long as the drives are properly stored.

Aside from using RAR's recovery record to test the files, you could also use regularly use "chkdsk /r" to force all sectors on the volume to be scanned, including unused ones, to try to head off any data loss by marking bad blocks. Manufacturer tools could also be used.

 

[MaxT2]

Thank you.

This question is not actually about the drive, it's about the OS. (...)

Yes, I had it in a corner of my head that this was a HDD and/or file system and/or OS question. That's why I mentioned in my first reply that the OSes that would read my archive drives are Win 10 Pro and Win 11 Pro (botch of which I expect to work the same way) and file system is NTFS. (I should add this to original post maybe.)

I would, barring a definitive answer, assume...

I'm assuming stuff since the beginning, I'm hoping to, at some point, stop assuming, thanks to this thread.

(00000001.chk for example), but depending on the data type, that might not do any good. Most of us have probably had chkdsk create a ton of "recovered" files from things like orphaned blocks but they're just parts of files and unusable.

That's why I pay much more attention to testing the RAR archives with RAR tools . Sometimes, chkdsk, Western Digital Dashboard, Victoria, HDD Scan and such tools don't detect anything, but yet some files are corrupted at what I would call the logical level (not sure if that name is accurate).

I never had any clue what to do with .chk files and the fact after decades of existence, chkdsk still doesn't tell what files are damaged is... disappointing.

Chkdsk and other tools can't reintegrate it into the original file with some zeroes to replace the bad data, giving the recovery record a chance to work. But chkdsk would probably try to create a recovery file containing the entire contents of the original RAR file, minus the bad data. Maybe the recovery record could work with that, but depends on just how big these archive files are. Would you have the space for chkdsk to do that?

Indeed, if chkdsk or such detect a problem at disk level, the first thing I do next is a full test at RAR level.
And indeed, it is better not to fill the drive up to 100% to leave some space for operations (such as repaired, or sometimes extract.)

So if the RAR recovery record can't deal with at least 4MB of damaged data, that might not be a good size to use (either use a smaller cluster size, or a larger recovery record). (4MB to account for damage that happens to pass from one allocation unit to another, last block of one to first block of another.)

Yeah, but that's because we're still assuming that my initial concern is based...

And of course, what happens if the bad blocks affect both some data and the recovery record? It would be like having the parity block in a RAID array get corrupted at the same time as the data. No way to rebuild without the full record.

You just made my brain run wild... What would happen to a RAR file is the recovery record was corrupted, independently from the fact that other data would be corrupt of not. (Maybe I should extend the size of my recovery records, and yet add .par2 files, and then put then all inside a container file, and then code/script some nested testing, until I'm left 10% of each drive dedicate to archive and 90% recovery data ;-) )

Maybe the best idea is archiving to two drives (RAID or separately),

I am archiving on 2 drivers and archiving the most important stuff on 3 copies but as I replied earlier, this is getting away from the initial question, which is focus on understanding one factor of the allocation unit size choice.

if it's that important. If you don't care that much or you're going to test these archives pretty regularly though, like every few months, one drive is probably fine. Consider the music industry which recently discovered their hard drives with stuff like master tracks from decades ago had gone bad because they hadn't been testing them or moving the data to newer storage.

Anyway, most of the remasters they made after 2000 are worth peanuts to me ;-) And so, I'm not surprise.
So people used to say I'm paranoid... but I see more and more run into catastrophes that I could have avoided..

Even a yearly test would probably be fine if the data isn't life-saving or financially-binding or something, as long as the drives are properly stored.

I do test the files at RAR level + SMART data + SMART short test (usually with Western Digital Dashboard) approximately once or twice per year.
And I run a LONG tests (again with Western Digital Dashboard or maybe Victoria or HDD Scan) less often, maybe every 1.5 year or so.

Aside from using RAR's recovery record to test the files, you could also use regularly use "chkdsk /r" to force all sectors on the volume to be scanned, including unused ones, to try to head off any data loss by marking bad blocks. Manufacturer tools could also be used.

I usually don't do that as I rely primarily on RAR archive testing, but it's a good remark regarding unused sectors.

 

[MaxT2]

If forgot this in my previous reply but someone made some study on how much RAR recovery record can recover based on various situations: https://www.liamfoot.com/analysing-the-effectiveness-of-winrar-s-rar5-recovery-records#heading_20

(I used to know some numbers about it and it was something like 1:1 or 1:2 ratio, but I can't remember and I'm not going into right now.)

 

[stuff and nonesense]

Context
Considering a large (rotating) hard drive. This drive will hold large *.rar archives files. (The files already exist and will be transferred from some of my older drives.)

*.rar files come with a "recovery record" (RAR equivalent of .par files) of usually 3% (sometimes I set 5%) which would allow them to get repaired in case they are corrupted. But if the damage on a given file is too large, it wouldn't be able to repair. (This always depends on the size allocated to recovery data).

From my understanding, a 2 MB Allocation Unit Size means that the system will read/write 2 MB at once on this drive.

Your question about 2MB allocation size, This is the minimum space taken up by an entry on the hard drive. A 2MB file will fill it, a 2B file will fill 2 bytes and there will be 1,999,998 byes of slack space. This is space that won’t be written to as the sector is used.
From what I can gather about RAR recovery tools they will recover what can be recovered from a corrupted RAR file but will not necessarily “fix” the corrupted bytes.
Assume that you don’t have a recovery method. If a few byes in a sector are corrupt then the file is corrupted. Multiple backups are needed to mitigate against the risk.

You have large RAR files, can they be broken up? Losing a few bytes in a collection of compressed files is less risk than losing a few bytes in a huge compressed file.

Relying on data recovery is risky. You are better insuring your data against loss by having 3 regularly refreshed backups as well as your computer’s working drives. 4TB of spinning rust is cheap these days.

 

[evermorex76]

I'd bet on the assumption being correct, with like 3 to 1 odds. It just doesn't make any sense for the OS itself during normal operation to attempt to do anything with a cluster of data where it can only read part of the cluster. The OS ought to report a file problem in Event Viewer (or other OS equivalent) and the application should fail, or report an error and give you the option to abort/retry/fail, or just considers it unimportant and keeps going, because the application is what cares about it. The OS tells the application "I can't read this cluster of the file" and some applications might give you an option (like disk cloning). You might even end up with a readable RAR file that's just missing some block of data where that 2MB would have been, since RAR and Zip and others compress blocks rather than the entire file as one. (A few frames missing in a video perhaps.) It might depend on the actual tool you use for opening it.

Is there a known quantity of recovery data that is recommended for being able to recover with specific amounts of corruption? You say sometimes you use 5%, so does that ensure you could recover if 10MB of data was lost, or 20? Or only 2? Anything concrete like that? (And then I refresh and see you answered that.)

Also if these are media they're probably already pretty well compressed, so you could just TAR the files rather than wasting time with compression, assuming you couldn't just use folder structure for some reason. I know that's off-topic.

 

[USAFRet]

Relying on data recovery is risky. You are better insuring your data against loss by having 3 regularly refreshed backups as well as your computer’s working drives. 4TB of spinning rust is cheap these days.

Absolutely.

I do not rely on any 'data recovery' to resurrect my data.
RAR, 3rd party company, whatever.

Defense in depth, with multiple copies on multiple drives.

 

[evermorex76]

Skimming quickly through the liamfoot.com article, I ran across this: "Note that in practice, a single byte of damage will corrupt an entire block of data sized up to 64k." (The block size that RAR works with.)

That presumably is referring to a drive using the default cluster size of 4k. But it points out that even when the OS CAN still read all the other sectors and different clusters, the RAR format will treat that entire 64k (16 clusters) as corrupt. Your 2MB clusters are going to have 32 of those RAR blocks inside of them, so if the OS can't read an entire cluster due to one bad sector, RAR is going to be trying to recover 32 of its own blocks with zero data. It might not necessarily be the number of bytes that makes a difference but the number of blocks that have to be recovered, and if those RAR blocks are completely missing, the recovery record might actually be useless if you've got such large clusters because so many blocks get lost.

Getting confusing using blocks for two things so I switched to sectors for the physical drive.

 

[MaxT2]

Your question about 2MB allocation size, This is the minimum space taken up by an entry on the hard drive. A 2MB file will fill it, a 2B file will fill 2 bytes and there will be 1,999,998 byes of slack space. This is space that won’t be written to as the sector is used.

Yes, I understand this. But I also thought that Windows would read data according to the allocation size (maybe that's not the case), and from the point of view, if a few are corrupted, it the whole sector impossible to read? Despite all kind of reply that "go around it", I still don't have the reply to that question..

From what I can gather about RAR recovery tools they will recover what can be recovered from a corrupted RAR file but will not necessarily “fix” the corrupted bytes.

RAR does not fix anything at disk level. It will most of the time create a copy of the file where it will have recomputed the inaccurate data from the content based on the recovery data, assuming that recovery data is big enough in regards the corrupt data quantity.

Assume that you don’t have a recovery method. If a few byes in a sector are corrupt then the file is corrupted. Multiple backups are needed to mitigate against the risk.

I do have multiple backups. That more or less out of topic regarding the initial question. Yet, they are not always needed, most times, I've been able to simply repair data using RAR recovery feature.

You have large RAR files, can they be broken up? Losing a few bytes in a collection of compressed files is less risk than losing a few bytes in a huge compressed file.

Indeed, the larger the file, the larger the recovery data and the more data it can recover. So in a sense, smaller files are more at risk if they get corrupted. On the other hand, statistically, smaller files are less likely to be where the corruption is. (I mostly have like 5 GB to 300 GB file and sometime bigger, but there are also some MB and kB files that I don't want broken.

Relying on data recovery is risky. You are better insuring your data against loss by having 3 regularly refreshed backups as well as your computer’s working drives.

I'm relying on both, as I already explained I have 3 copies for the most important stuff and 2 for the less important (because of budget), I'm aware of the the risk and again, it's out of the topic.

I'm not asking for people to explain how to manage my whole archive workflow, especially with stuff I already know. I'm just asking is some bytes are corrupted, is the whole sector corrupted. And now one has the answer so far.

4TB of spinning rust is cheap these days.

I won't go far with, I currently have something between 50 TB to 75 TB...

 

[MaxT2]

I do not rely on any 'data recovery' to resurrect my data.
Defense in depth, with multiple copies on multiple drives.

I never said I rely on 'recovery only', and I think it's much better to rely on both.

 

[USAFRet]

I'm just asking is some bytes are corrupted, is the whole sector corrupted.

Basically, yes.

It is considered corrupted.

The OS will no longer read that sector. And the drive firmware will (hopefully) remap that sector to one of its unused ones.

Expecting the OS to read a partial sector and ignore the bad "bytes" goes against the whole concept of the sector size concept.

 

[MaxT2]

Skimming quickly through the liamfoot.com article, I ran across this: "Note that in practice, a single byte of damage will corrupt an entire block of data sized up to 64k." (The block size that RAR works with.)

That presumably is referring to a drive using the default cluster size of 4k. But it points out that even when the OS CAN still read all the other sectors and different clusters, the RAR format will treat that entire 64k (16 clusters) as corrupt. Your 2MB clusters are going to have 32 of those RAR blocks inside of them, so if the OS can't read an entire cluster due to one bad sector, RAR is going to be trying to recover 32 of its own blocks with zero data. It might not necessarily be the number of bytes that makes a difference but the number of blocks that have to be recovered, and if those RAR blocks are completely missing, the recovery record might actually be useless if you've got such large clusters because so many blocks get lost.

Getting confusing using blocks for two things so I switched to sectors for the physical drive.

Thanks. That's the closest thing to the answer we got so far.
The block/sector thing is indeed confusing.
Maybe that means the optimal allocation size when using RAR would be 64k ? I guess that RAR knows what to do if it received 2 MB from the OS and can split it to 64K blocks in memory without re-reading the same data.
Yet, we should check if that part is talking about RAR4 or RAR5 because RAR4 is still often talked about but is quite superseeded by RAR5.

That part of text also seems to implied that, even if it's partly corrupted, the OS would actually read the full sector... If I get it right...