Question Windows 10 Spanned Volume Suddenly Missing (Dynamic Disks: "Invalid")

Page 2 - Seeking answers? Join the Tom's Hardware community: where nearly two million members share solutions and discuss the latest tech.
ISTM that one critical structure is still missing. AFAICT, the initialisation process zeroed out the PRIVHEAD sector (the first sector) in the LDM Metadata partitition. Everything else in that partition seems to have been left untouched. Fortunately there are two backup copies of PRIVHEAD near the end of the metadata. ISTM that this PRIVHEAD copy should be reinstated by copying it to sector 260130.

However, I still think there is more to do. There appear to be 3 GUIDs in PRIVHEAD, and I expect that these should match the GUIDs in sector 2. I'm still studying this, though. The earlier LDM reference I linked to appears to be for MBR based dynamic volumes, not GPT, so we need to keep this in mind.
 

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Unfortunately that didn't cut it, though it did have some kind of effect. The drives now show up like this:




I uploaded the current state of the LDM data to the same folder I mentioned previous if you want to double check it.

Tomorrow or Wednesday I'll see if I can get that other span going going so we can see if there is anything made clear from a working example. If not, I'll just have to figure out where I'm going to dump all the data one way or another.
 

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Moving your data to a safe space offline should be the first thing, not 'figure out later'.
Yes of course, I stated this in the op. Unfortunately I can't afford the amount of storage that would be required to backup what im storing on these drive at the moment. In time I will be able to, and ive known the whole way I've been taking a risk until then. That's just the way it is.
 

USAFRet

Titan
Moderator
Mar 16, 2013
111,875
1,606
154,440
18,453
Yes of course, I stated this in the op. Unfortunately I can't afford the amount of storage that would be required to backup what im storing on these drive at the moment. In time I will be able to, and ive known the whole way I've been taking a risk until then. That's just the way it is.
I get that.
The comment was also for other people who will come across this thread in years to come.
 
Reactions: oblivioncth
Your results are as I expected. The data partitions are in the right place, but the LDM Metadata partition needs to be edited. There is a sector with a PRIVHEAD signature near the end of each 260130_2048 dump. This copy needs to be restored to sector 260130 (offset 0 in the dump).

You can see an example of the PRIVHEAD sector here:

https://stackoverflow.com/questions/8427372/windows-spanned-disks-ldm-restoration-with-linux

Note that DMDE has a Copy Sectors command, but be careful when specifying the destination.
 

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Your results are as I expected. The data partitions are in the right place, but the LDM Metadata partition needs to be edited. There is a sector with a PRIVHEAD signature near the end of each 260130_2048 dump. This copy needs to be restored to sector 260130 (offset 0 in the dump).

You can see an example of the PRIVHEAD sector here:

https://stackoverflow.com/questions/8427372/windows-spanned-disks-ldm-restoration-with-linux

Note that DMDE has a Copy Sectors command, but be careful when specifying the destination.
Gah, didn't seem to do anything. To confirm, I copied the backup of the PRIVHEAD sector from each respective drive at sector 262177 to sector 260130 of each respective drive. I did it directly using HxD to ensure I was using the correct destination. I wonder if it has to do with GUIDs no longer matching. Hoping that a working example proves useful.

EDIT:
I'm not sure about how much of this applies to GPT based partitions but here is some nice info on the PRIVHEAD Sector: https://www.apriorit.com/dev-blog/345-dynamic-disk-structure-parser

The FILETIME for the sector on Drive 4 converts to "Sunday, July 28, 2019 4:05:40am " which sounds about right for when I originally created the volume.

I was able to reorder some things I'm doing so I went right for the drives. I found two old 100GB sata drives that I'm going to make a spanned volume across and then examine them and dump all of the same primary sectors that we've lifted off of my borked drives.

EDIT 2:
You may have checked this information already, but...
Code:
ALL LDM DATABASE DATA IS BIG ENDIAN

DRIVE 4 @ 260 130

struct PrivHead
{
    uint8_t magic[8]; = "PRIVHEAD"
    uint32_t checksum; = 0x0000310F
    uint16_t major; = 0x0002
    uint16_t minor; = 0x000C
    uint64_t timestamp; = 0x01D544F9B828243F‬ ->  Sunday, July 28, 2019 4:05:40am (https://www.silisoftware.com/tools/date.php)
    uint64_t sequenceNumber; = 0x0000000000000003
    uint64_t primaryPrivateHeaderLBA; = 0x00000000000007FF
    uint64_t secondaryPrivateHeaderLBA; = 0x0000000000000740
    uint8_t diskId[64]; = "e8aa8e70-e6b3-b94c-97e0-46945532e999............"
    uint8_t hostId[64]; = "1b77da20-c717-11d0-a5be-00a0c91db73c............"
    uint8_t diskGroupId[64]; = "cf3cc96b-1b67-bc4e-ac4c-7fbece90ad40............"
    uint8_t diskGroupName[31]; = "TrueImageDg0............" (Strange size, 31 bytes)
    uint32_t bytesPerBlock; = 0x00000200
    uint32_t privateHeaderFlags; = 0x00000000
    uint16_t publicRegionSliceNumber; = 0x0000
    uint16_t privateRegionSliceNumber; = 0x0000
    uint64_t publicRegionStart; = 0x0000000000040022/262 178 <- THIS IS START SECTOR ON DISK 3!
    uint64_t publicRegionSize; = 0x00000003A37D2A6D/15 627 790 957 <- LARGER THAN THE REPORTED/CALCULATED SIZE FOR BOTH DISKS
    uint64_t privateRegionStart; = 0x000000000003F822/260 130 <- Start sector we used for this drive (Disk 4)
    uint64_t privateRegionSize; = 0x0000000000000800/2048
    uint64_t primaryTocLba; = 0x0000000000000002
    uint64_t secondaryTocLba; = 0x00000000000007FD/2045
    uint32_t numberOfConfigs; = 0x00000001
    uint32_t numberOfLogs; = 0x00000001
    uint64_t configSize; = 0x00000000000005C9/1481
    uint64_t logSize; = 0x00000000000000E0/224 (Thought this would be 64MB, must be another log)
    uint8_t diskSignature[4]; = 0x0FC7D4CF/264754383/".ÇÔÏ"
    Guid diskSetGuid; <- Unknown, data after diskSignature is all 00
    Guid diskSetGuidDupicate; <- Unknown, data after diskSignature is all 00
};


DRIVE 3 @ 260 130

struct PrivHead
{
    uint8_t magic[8]; = "PRIVHEAD"
    uint32_t checksum; = 0x000031AE
    uint16_t major; =  0x0002
    uint16_t minor; = 0x000C
    uint64_t timestamp; = 0x01D544F9B828243F ->  Sunday, July 28, 2019 4:05:40am (https://www.silisoftware.com/tools/date.php)
    uint64_t sequenceNumber; = 0x0000000000000003
    uint64_t primaryPrivateHeaderLBA; = 0x00000000000007FF
    uint64_t secondaryPrivateHeaderLBA; = 0x0000000000000740
    uint8_t diskId[64]; = "e6a400ac-b8bd-4b48-a53d-a3832ff994b7............"
    uint8_t hostId[64]; = "1b77da20-c717-11d0-a5be-00a0c91db73c............" <- Matches the ID on Disk 4
    uint8_t diskGroupId[64]; = "cf3cc96b-1b67-bc4e-ac4c-7fbece90ad40............" <- Matches the ID on Disk 4
    uint8_t diskGroupName[31]; = "TrueImageDg0............" (Strange size, 31 bytes)
    uint32_t bytesPerBlock; = 0x00000200/512 
    uint32_t privateHeaderFlags; =0x00000000
    uint16_t publicRegionSliceNumber; = 0x0000
    uint16_t privateRegionSliceNumber; = 0x0000
    uint64_t publicRegionStart; = 0x0000000000040022/262 178 <- Start sector we used for this disk
    uint64_t publicRegionSize; = 0x00000003A37D2A6D/15 627 790 957 <- LARGER THAN THE REPORTED/CALCULATED VOLUME SIZE FOR BOTH DISKS. Matches Disk 4
    uint64_t privateRegionStart; = 0x000000000003F822/260 130 <- MISMATCH BETWEEN SECTOR START WE USED FOR THIS DISK (publicRegionStart), AND SECTOR START WE USED FOR DISK 4 (privateRegionStart)
    uint64_t privateRegionSize; = 0x0000000000000800/2048
    uint64_t primaryTocLba; = 0x0000000000000002
    uint64_t secondaryTocLba; = 0x00000000000007FD/2045
    uint32_t numberOfConfigs; = 0x00000001
    uint32_t numberOfLogs; = 0x00000001
    uint64_t configSize; = 0x00000000000005C9/1481
    uint64_t logSize; = 0x00000000000000E0/224 (Thought this would be 64MB, must be another log)
    uint8_t diskSignature[4]; = 0x0FC7D4CF/264754383/".ÇÔÏ"
    Guid diskSetGuid; <- Unknown, data after diskSignature is all 00
    Guid diskSetGuidDupicate; <- Unknown, data after diskSignature is all 00
};
 
Last edited:

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Could it be that the sector ranges I entered at the beginning and end of the Drive should have been in Big Endian format? That had more to do with the GPT standard part of the file structure rather than the LDM, though it was under the LDM entry section, but its a thought.

More info:
* Note * - All offset regions declared after "privateRegionStart" are based upon it (privateRegionStart = 0x00 relatively speaking. i.e. "log" start is actually (260 130 + 1498) *512 = Offset 0x7FBF800‬ -> Sector 261,628: Signature "KLOG"

Described in more detail in that apriorit.com (if we really need them I may try to build those tools, this documentation seems excellent).

Code:
TOCBLOCK (SAME ON BOTH DISKS)

struct TocBlock
{
    uint8_t magic[8]; = "TOCBLOCK"
    uint32_t checksum; = 0x000008B8
    uint64_t updateSequenceNumber; = 0x0000000000000003
    uint8_t zeroes[16]; = 0x0000000000000000 (duh)
    TocRegion config; = SEE BELOW
    TocRegion log; = SEE BELOW
};


struct TocRegion (CONFIG)
{
    uint8_t name[8]; = "config.."
    uint16_t flags; = 0x0000
    uint64_t start; = 0x0000000000000011
    uint64_t size; = 0x00000000000005C9/1481
    uint16_t unk1; = 0x0006 (Unknown?)
    uint16_t copyNumber; = 0x0001
    uint8_t zeroes[4]; = 0x00000000 (duh))
};

struct TocRegion (LOG)
{
    uint8_t name[8]; = "log....."
    uint16_t flags; = "0x0000"
    uint64_t start; = 0x00000000000005DA/1498
    uint64_t size; = 0x00000000000000E0/224
    uint16_t unk1; = 0x0006 (Unknown?)
    uint16_t copyNumber; = 0x0001
    uint8_t zeroes[4]; 0x00000000 (duh))
};
I needed to back up the disks I was going to use for the working example spanned volume and one of them has a ton of bad sectors and cant be read well so i need to see if I have another. I think I may.

I also notice when I edit my posts too many times (damn typos) it gets hidden for moderator approval so sorry if our conversation has been a bit "hole filled" because of that. I think I may have missed part of one of your posts for the same reason.

EDIT:
It is worth noting that at least in the Partition Table MBR/GPT and FAT/FAT32/NTFS Bootrecord modes, DMDE doesn't seem to read the PRIVHEAD sector correctly. Also, I found another drive to use that seems good. First drive is already backed up (they have old stuff on them that a family member wants backed-up just in case) and zero'ed, second one is zeroing now. I will set up the volume tomorrow.

EDIT 2:
I built the command-line tool that was provided in that link. It isn't detecting either (or both together) drives as LDM volumes which I find interesting (is it simply a bug/regression? Or is it not detecting them as such because of what we are missing?) and I am stepping through the source to see if I can figure out why.

As for the example volume, both drives are ready to go, I'm just updating the machine to Windows 10 (been needing to for a while anyway) so that both machines are on the same OS, just on the off chance that they generate spanned volume slightly differently.

EDIT 3:
Ah, as I feared the tool is only able to interpret MBR disks. It seems the first thing the program tries to do after it determines a drive is an LDM based volume is read the PRIVHEAD and everything offset after that is based on what the PRIVHEAD contains. So, I'll see if I can just hard code in the correct sector for where the PRIVHEAD is on these drives and see if the rest works out.

I also found this: https://docs.microsoft.com/en-us/sysinternals/downloads/ldmdump , but it returns "Disk does not have LDM database", and since it says it runs on Vista and higher, I'm guessing it too is expecting an MBR disk. This one has no source so unfortunately its a dead end.
 
Last edited:

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Phew, man still lots to investigate, but I have the working example span setup and have dumped the bulk of the respective sectors that we have looked at on the corrupted span. This was done useing the same software I used to make 'Archive'. Since the drives were zeroed before hand I want to make sure I get everything that's on the drive since its 99% metadata (though there seems to be a few records of files on there that I haven't dumped yet, but they seem to relate to system volume information files like the $Logfile). What is super annoying is that no hex editor I can find nor DMDE have a function to advance to the next sector that doesn't contain just null bytes ('00') (seriously!?), so I had to manually scroll really fast near the beginning and end of each drive to get the data I have so far. I'm going to examine how the LdmParser accesses the raw disk and see if I can't use that API to write a quick command-line tool to do just that: You input a sector and it searches for the next sector of non-null data after that sector offset and allows you to do so until you hit the end of the drive.

TBH I can't believe this isn't a common feature as it is the only way I can ensure I've found every bit of data on a mostly empty disk. I supposed developers figure that most of the time you're working with disks that aren't primarily empty and that there is so much data in that situation that this function wouldn't be that useful; however, in this case such a function would be incredibly useful.

For now, I have uploaded the dumps I have of the working example along with a bit of info about the drives to the Google Drive folder. This time I set the files names in a way so that the dumps are in the same order that they appeared on the disk, which should make thinking through offsets a lot easier. Unfortunately, because that one drive had bad sectors and I had to use another they drives for this span aren't the same size (100GB vs 300GB) but I don't think that should matter.

The start and end sectors in the GPT Entries sector are definitely not supposed to be Big endian. Also, interestingly there is a discrepancy between the start sectors of the working example: In the example, the 3rd GTP entry on both drives (L D M d a t a p a r t i t i o n) has sector 262 178 marked as the start, while for the 'Archive' drives, drive 3 also has sector 262 178 marked as the start, while drive 4 as we know is different and has sector 264 192 marked as the start. 262 178 is also the value of "publicRegionStart" in the PRIVHEAD sector for drive 3 AND 4. This may just be a left over property of the fact that this disk was once Basic and was then converted to Dynamic and not actually mean anything, but it may very well be a mismatched value between the GPT LDM entry and the PRIVHEAD value. Finally, both Drive 4 and Drive 3 of 'Archive' are missing their protective MBR sectors. While those are important for programs accessing the drive, I'm not so sure this would have any effect on modern Windows' ability to read the drives correctly.

EDIT:
Got a test program that can read 512 byte sectors directly from the selected disk working so I'm going to make that utility I mentioned.
 
Last edited:

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Took forever because of an annoying issue with the OS reporting the disk sizes incorrectly but I finished the utility. I will now finish dumping all of the useful data from the working example volume and hopefully find any significant differences other than the few I listed already.

The tool is in the drive folder if it ends up being useful to anyone.
 
I'm thinking that the structure of the LDM Data partition may include a header from sectors 262178 to 264192, in which case we probably should not have edited sector 2.

I would have thought that LBA 34 would be the primary PRIVHEAD sector, but you don't appear to have found one??

BTW, I agree that searching for the next non-zero sector would be a useful feature. I often need to do this.

Elsewhere you mentioned the protective MBR. AISI, it should be present on both drives.
 

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Any more progress? If not, I'll try to analyse each of the data structures, if only to understand them better.
Been a tad busy, but I've become enthralled with trying to fix this volume manually now that I'm starting to understand the involved formats.

1) I started a doc in the root of the folder with a list of known potential issues that I (and feel free to yourself, more on that in (2) ) will update as we come upon more discrepancies or dig through details.

2) If you want to add/edit anything to/in the Drive folder I can PM you a share link with editing permissions or you can PM me a google account to share it with directly. If you'd rather just keep notes to yourself and post the significant ones here that's fine as well.

3) The utility I wrote had a couple bugs from a single conversion from unsigned to signed integers somewhere so I squashed that and uploaded the new version. It could use some polish and optimization but works really well and allowed me to make sure I combed over the entirety of each drive

4) With the help of the tool I grabbed a couple more dumps from the working example. Everything in the '0' and '1' 'Raw Data' folders should be everything on those drives that matters for their operation. For drive '1' I believe those dumps are actually every non-null section of data on the entire drive, aside from a couple patches of data that were just all high (0xFF) and didn't seem to be metadata or structure related. For drive '0' it is pretty much the same deal except that I did omit some sections of data that seemed to be files inserted by the OS that were well into the data section of the volume and didn't seem at all to contain any info relevant to the construction of the volume. I loosely marked around where that section of data was in case it is ever needed. There are also a few dumps of what also seems to be straight data but its stuff that occurred early in the volume and some of it looks like it could be hidden $SystemVolumeInformation files so I included them in case.

5) There is a file editor for a proprietary format in some old game I've been working on in my spare time before this happened and I've been bouncing back to that here and there to keep my mind fresh on what I was last doing with it. Otherwise, I now will start analyzing the dumps as well and probably focus on trying to make a bare-bones map with all the markers for where each section of the 'Working Example' span starts/ends so that we have something to compare to for the damaged volume. I may write a short utility that works in tandem with the other one that can lightly (mainly just break up) parse the key sectors we've discussed (GPT entries, PRIVHEAD sector, etc.) using the references that have accumulated in this thread, so that again, comparing between the example and corrupted spans is easier.

6) Yes, something is definitely up with sectors 262178 to 264192 on Drive 4 so I'll be sure to examine that as well in relation to the example and Drive 3.

7) The location of the PRIVHEAD sectors on the example span confused a lttile me as well since in someways their location was fairly dissimilar to the ones found in Drives 4 and 3, but hopefully the reason for this will be made clear once the offsets in each record are checked and compared and these differences could very well be part of the problem with the damaged span. The only thing near LBA 34 is the table of contents sector at LBA 36 on both Drive 0 and 1 of the example (I double checked). Otherwise the only PRIVHEAD blocks present were the ones at the locations you can see in the dumps I provided. I can search for them specifically in case if this proves to be a significant issue in terms of making sense of the dumps.

8) I will look them over and a do a little bit of research, but I have a feeling that the protective MBR sector employed in GPT based disks is not partition specific and I can just transplant the one from the example into Drive 3 and 4.
 
Last edited:

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
I'm assuming you expecting the PRIVHEAD sector to be much earlier than it is in the working example is based on this: http://www.ntfs.com/ldm.htm

Unfortunately it seems that this is only for MBR (as you mentioned could be an issue multiple times), which can be confirmed by the fact that their diagram shows the LDM metadata occurring at the end of the drive. This is commented on in the wiki article for LDM:
On a disk partitioned with the MBR Partition Table scheme, the Logical Disk Manager metadata are not stored in a partition, but are stored in a 1 MiB area at the end of the disk that is not assigned to any partition.
In the GPT version of LDM there is an LDM Metadata partition that is the first partition on the disk and is hidden. This contains a table of contents (TOCBLOCK) and holds the same data that would be at the end of the disk in the case of MBR. Somehow this points to the PRIVHEAD sector that is located deeper in the disk. I'm not sure exactly where yet and am trying to find info about this metadata parition but it seems that the GPT variant of LDM does not have nearly as much documentation online as the MBR version does.

I've mapped out a good amount of the working example under the document "Span Map" that is within the "Working Example folder on Drive, but still have a fair bit to go. I'm really hopping to find documentation on the GPT LDM Metadata partition so I can be more certain of things. For example, the GPT entry for the first partition on drive 0 says the first sector is LBA 34, but the TOCBLOCK for that partition resides at LBA 36. It would be nice to know if the first two sectors are known to be blank or reserved and/or have a name, instead of just noting those two sectors down as "NULL Data".

Lastly, this tool might come in handy as a last resort, or potentially for documentation reasons depending on how much information it parses. It seems fairly capable at a glance: http://www.rodsbooks.com/gdisk/
 
Last edited:

oblivioncth

Honorable
Sep 5, 2012
33
1
10,535
0
Ok,

It took quite a while but I have finally finished completely mapping and parsing the metadata by hand of the 'Working Example' span. The process was very enlightening and I now have a much better understanding of how Windows handles dynamic disks in the GPT format. It is now very clear how the LDM meta-data partition is laid out and I found a lot of clues as to what might be up with with the 262 178 vs 264 192 issue on Drive 4.

I have reverted the changes we made to the GPT entries and deleted the "restored" PRIVHEAD copy and will now begin the same mapping process for the broken 'Archive' span, which hopefully won't take as long since I can copy and pate some things and am familiar with how the sections relate to each other.

The file "Span Map" in the "Working Example" folder has all of the details, complete with a full overview map that features links to each worksheet for every metadata subsection. There are also plenty of helpful notes for many of the entries.

Given this now clear and labeled data, as long as the VMDB on both Drive 4 and 3 are in-tact and original, as in not overwritten/corrupted by the initial damage OR anything windows might have done after the drives partially came back online after the changes we had made (Windows did make some changes to the GPT partition tables on drive 4 after I restarted that were incorrect, so it is a good thing we have backups in the forms of the dumps I've made), as recreating those hundreds of blocks by hand would be nearly impossible and take way, way to long, I am confident that the 'Archive' span can be repaired successfully. I say this because now that I've been through it all, other than some potential small issues with some IDs (there some 4-byte ones aside from the 16-byte GUIDs that I'm a little uncertain of) everything but the VMDB VBLK entries are easily manipulated and straightforward.
 

ASK THE COMMUNITY

TRENDING THREADS