Protecting and Archiving Data Files on M-Disc BD-R

[mikeysp]

For storing data to M-Discs and protecting the data from being deleted. How do I do this?

Additional details:
1. I currently have Adobe Encore and IMGBurn as DVD authoring software. I am ready and willing to purchase better software if it makes sense.
2. My burner is the pioneer blu ray burner BDR-209DBK
2. I will be using BD-R M-Discs between 25-100GB, (Not sure which size yet).
4. If possible, I desire to finish burning the data to disc, so no one can lose the data, short of damaging the disc physically. Protection from deleting files, reformatting the disc, etc.. if possible, or as close to this as is sensible.
5. If I can do the above, and it only uses half the storage space on the disc, will I be able to go in a month later and use the unused storage space for additional files? If not, no problem, I will just burn when I am near capacity.
6. What is the recommended max amount (%) of disc space to use to keep data from getting corrupted?

I have read for a couple of hours searching the internet, but I am left confused.

Thank you for your consideration.

Michael.

 

[Spectre694]

Your thinking too hard they are just fancy (higher quality needs a more powerful laser) DVD/BD so most all is the same between them.

1) shouldn't matter so long as it writes to BD the difference between mdisk and the normal ones is taken care of by the drive
2)listed as compatible
3)doesn't matter
4) they are BD-R not RW's once they are burned that is it can't change them
5)If they are formatted as UDF not ISO then yes.
6)Doesn't matter

All that said are M-Disks really the best choice for your situation? They are expensive archival media generally suited for write once maybe read later use.

 

[mikeysp]

First, thank you very much for such a helpful answer. I am indeed thinking too hard

You asked: "All that said are M-Disks really the best choice for your situation?"

We are trying to store large (20-80GB possibly) data raw video AVCHD and MPEG 2 and additional files to set on a shelf for 200-300 years with a desire for that generation to have access. We certainly have thought: in another decade, there will be some other technology that makes the M-Disc obsolete; however we want to operate on the assumption, the discs will go on the shelf this year, and will be forgotten for those 200 years. Thus, we thought M-disc would be the the best option. Especially considering we already have the M-Disc comparable burner.

BTW, your answer led to another question.... When it comes to museum archival scenarios, which is better: UDF or ISO?

I would greatly appreciate your advice Spectre694?

 

[Solandri]

It is impossible to delete or overwrite files in write-once media like CD-Rs, DVD-Rs, and BD-Rs. (RW are another matter.)

When you write to a BD-R for a second time, what's actually happening is the new copy of any changed files + any new files are written to the disc, then a new directory listing is written. When you read the disc, it ignores any earlier directory listings and uses the latest one instead.

If a file has been "deleted", its entry is simply missing from the newest directory listing.

If a file has been "overwritten", a new copy of the file is stored in the second burn, and the newest directory listing points to that new copy.

For files unchanged since the first burn, the newest directory listing points to their original location in the first burn.

Even though the original file has been "deleted" or "overwritten", it is still present on the disc. There is software which lets you access previous directory listings and read the older copies of those files. i.e. You can tell it to read the BD-R up to the first burn, or the second burn, etc. In that respect, you never need to worry about deleting, overwriting, or reformatting a BD-R, DVD-R, or CD-R. The data is still there. Short of physical damage or deterioration to the disc, there is no way to lose data from write-once media.

When you finalize or close a disc, you're basically writing a flag to it that tells the computer that no more data is allowed to be written to this disc. So once you finalize it, it becomes read-only media. You can never write to it again. Any remaining unused space is wasted. It does not provide any additional protection for the data on the disc, other than making it easier to access said data without the specialized software I mentioned earlier. (i.e. Someone can't write a blank directory listing to the disc making you think it's an empty disc from a bad burn, and trick you into throwing it away.)

Given the expense of M-Discs, I would recommend burning files to BD-R or DVD-R as an interim backup. Once you're sure the file is never gonna change again, then burn it to M-Disc. If you burn the interim versions to M-Disc, you're going to fill it up with multiple copies of the file even though you're only interested in archiving the last one. (OTOH, if you want to archive the editing history of the file, then burning these interim versions to M-Disc would accomplish just that.)

 

[Spectre694]

First, thank you very much for such a helpful answer. I am indeed thinking too hard

You asked: "All that said are M-Disks really the best choice for your situation?"

We are trying to store large (20-80GB possibly) data raw video AVCHD and MPEG 2 and additional files to set on a shelf for 200-300 years with a desire for that generation to have access. We certainly have thought: in another decade, there will be some other technology that makes the M-Disc obsolete; however we want to operate on the assumption, the discs will go on the shelf this year, and will be forgotten for those 200 years. Thus, we thought M-disc would be the the best option. Especially considering we already have the M-Disc comparable burner.

BTW, your answer led to another question.... When it comes to museum archival scenarios, which is better: UDF or ISO?

I would greatly appreciate your advice Spectre694?

Not a problem and yes for your use case it does actually seem like M-Disk is a valid choice. A important part of long term archival storage though that most people forget is that the format and common knowledge for it may disappear. So anything like that should both be stored with a way to retrieve the data (small comp with a BD drive) and instructions on retrieval plus details on what it is(media drive comp etc.) in case your method of retrieval doesn't work after setting there for so long.

Ex. you had some data on 8 inch floppies (1970 ish) That is only around 40 years ago (much less than your time frame of 200 years)and it would already be pretty hard to find someone who knew what it was much less how to get a PC and drive to read them.

For UDF vs ISO that's a good question: [clarify first for ISO I mean joliet (9660) and UDF I mean (ISO 13346)]
I would say UDF (standard replacing joliet compatibility going forward is looking pretty good) in plain (single burn) It would give you one shot to burn the disk and once closed it can no longer have any changes made to it. VAT mode (what solandri described) lets you burn more to it or "delete" but once finalized (properly) it can no longer be changed. So a valid choice there too.

 

[mikeysp]

Thank you for the clarification Solandri! Yes, this will be our Archival medium, so the M-Disc will be used.

In light of your answer, I have another question: What software can access previous directory listings? I did some online searching, but was having trouble finding a solution.

It is impossible to delete or overwrite files in write-once media like CD-Rs, DVD-Rs, and BD-Rs. (RW are another matter.)

When you write to a BD-R for a second time, what's actually happening is the new copy of any changed files + any new files are written to the disc, then a new directory listing is written. When you read the disc, it ignores any earlier directory listings and uses the latest one instead.

If a file has been "deleted", its entry is simply missing from the newest directory listing.

If a file has been "overwritten", a new copy of the file is stored in the second burn, and the newest directory listing points to that new copy.

For files unchanged since the first burn, the newest directory listing points to their original location in the first burn.

Even though the original file has been "deleted" or "overwritten", it is still present on the disc. There is software which lets you access previous directory listings and read the older copies of those files. i.e. You can tell it to read the BD-R up to the first burn, or the second burn, etc. In that respect, you never need to worry about deleting, overwriting, or reformatting a BD-R, DVD-R, or CD-R. The data is still there. Short of physical damage or deterioration to the disc, there is no way to lose data from write-once media.

When you finalize or close a disc, you're basically writing a flag to it that tells the computer that no more data is allowed to be written to this disc. So once you finalize it, it becomes read-only media. You can never write to it again. Any remaining unused space is wasted. It does not provide any additional protection for the data on the disc, other than making it easier to access said data without the specialized software I mentioned earlier. (i.e. Someone can't write a blank directory listing to the disc making you think it's an empty disc from a bad burn, and trick you into throwing it away.)

Given the expense of M-Discs, I would recommend burning files to BD-R or DVD-R as an interim backup. Once you're sure the file is never gonna change again, then burn it to M-Disc. If you burn the interim versions to M-Disc, you're going to fill it up with multiple copies of the file even though you're only interested in archiving the last one. (OTOH, if you want to archive the editing history of the file, then burning these interim versions to M-Disc would accomplish just that.)

 

[mikeysp]

Thank you for answering my questions Spectre694. After a few tests and the answers from this discussion, my plan is to burn files to the M-Disc like a USB flashdrive, and drag the archive files onto the M-Disc as I need it. Once it is full, finalize the disc with ImgBurn. It will all be in UDF format.

This done, I found that you cannot delete the files, you cannot re-format the disc either. Just what I wanted.

First, thank you very much for such a helpful answer. I am indeed thinking too hard

You asked: "All that said are M-Disks really the best choice for your situation?"

We are trying to store large (20-80GB possibly) data raw video AVCHD and MPEG 2 and additional files to set on a shelf for 200-300 years with a desire for that generation to have access. We certainly have thought: in another decade, there will be some other technology that makes the M-Disc obsolete; however we want to operate on the assumption, the discs will go on the shelf this year, and will be forgotten for those 200 years. Thus, we thought M-disc would be the the best option. Especially considering we already have the M-Disc comparable burner.

BTW, your answer led to another question.... When it comes to museum archival scenarios, which is better: UDF or ISO?

I would greatly appreciate your advice Spectre694?

Not a problem and yes for your use case it does actually seem like M-Disk is a valid choice. A important part of long term archival storage though that most people forget is that the format and common knowledge for it may disappear. So anything like that should both be stored with a way to retrieve the data (small comp with a BD drive) and instructions on retrieval plus details on what it is(media drive comp etc.) in case your method of retrieval doesn't work after setting there for so long.

Ex. you had some data on 8 inch floppies (1970 ish) That is only around 40 years ago (much less than your time frame of 200 years)and it would already be pretty hard to find someone who knew what it was much less how to get a PC and drive to read them.

For UDF vs ISO that's a good question: [clarify first for ISO I mean joliet (9660) and UDF I mean (ISO 13346)]
I would say UDF (standard replacing joliet compatibility going forward is looking pretty good) in plain (single burn) It would give you one shot to burn the disk and once closed it can no longer have any changes made to it. VAT mode (what solandri described) lets you burn more to it or "delete" but once finalized (properly) it can no longer be changed. So a valid choice there too.

 

[Solandri]

In light of your answer, I have another question: What software can access previous directory listings? I did some online searching, but was having trouble finding a solution.

Anything that will let you read individual multisessions.

http://www.isobuster.com/multisessions.php

 

[hang-the-9]

We certainly have thought: in another decade, there will be some other technology that makes the M-Disc obsolete; however we want to operate on the assumption, the discs will go on the shelf this year, and will be forgotten for those 200 years. Thus, we thought M-disc would be the the best option. Especially considering we already have the M-Disc comparable burner.

In 200 years chances are pretty good there will be no drives to read your disks around, unless you store drives along with the disks and test them once every few months to make sure they are working for the next 200 years. Even now Apple is not including an optical disk in most of their systems.

 

[Solandri]

In 200 years chances are pretty good there will be no drives to read your disks around, unless you store drives along with the disks and test them once every few months to make sure they are working for the next 200 years. Even now Apple is not including an optical disk in most of their systems.

Optical discs are by far the longest-lived media. CDs made in the 1980s and CD-Rs from the early 1990s are still readable in modern Blu-ray drives.

HDDs have gone from ST/ESDI to IDE to SATA. On the server front, they've gone through several iterations of SCSI (not always backwards compatible) to SAS.

Ejectable disks have gone from 5.25" floppies to 3.5" to Zip drives (my graduate thesis is on a Zip disk somewhere) to obsolete.

Tape drives have gone through more formats than I care to count. On the non-computer front alone, there's been reel-to-reel, 8-track, audio cassette (the longest-lived at over 40 years), Beta, VHS, VHS-C, Hi8.

Ejectable flash drives have gone from PCMCIA (still have one in my old HP 200LX), to CF (a reduced length PCMCIA card) to SD to mini SD to microSD.

USB flash drives are about to go from USB-A to USB-C.

None of these are backwards compatible sans adapter. Except optical discs. Pop in a 1984 CD into a 2016 Blu-ray drive and it'll play.

I have no idea if there will still be optical drives which can read them in 200 years, but they're closing in on 35 years right now. Which makes them by far the best bet for archival data storage. The industry seems to have settled on 12cm as a handy size which still stores a good amount of data, so all future optical drives should be the same size and thus able to read older discs. (And don't say just copy the data to a new external HDD every 5 years. You've obviously never heard of bit rot. My NAS runs ZFS to help counter bit rot - random bit flips in the magnetic media. But the real drawback of HDDs is that they're not write-once. I have to regularly backup my NAS in case I accidentally delete or overwrite a file that I mean to be archiving. You don't have that problem with archival write-once optical media.)