mdadm / RAID grow mess

sammm00

Reputable
Jan 17, 2016
4
0
4,510
I'm in a bit of a mess with mdadm and my RAID array, I'd be very grateful if anyone had any ideas.

I tried to grow my RAID 6 array consisting of 5x 3TB drives with another 3TB drive, however after 12 hours the reshape didn't appear to be making any

progress; reshape = 0.0% (0/2930265088). Unfortunately at some point I suffered power loss and upon powering on the array wouldn't start. After

googling I've read that the first moments of a reshape are critical, however as the reshape seemed to be stuck I'm hoping there's a way to recover

from this? I also read that you can define a back up file, unfortunately the guide I was following for the reshape didn't mention this, so I don't

have one. All 5 are reporting clean and correct checksums.

At the moment I don't care about trying to grow the array, I'd just like to try and recover data if possible.

Here's the output of mdadm --examine for each of the old drives.
Code:
root@nas:~# mdadm --examine /dev/sda
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
           Name : nas:Storage  (local to host nas)
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 11721060352 (11178.07 GiB 12002.37 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=944 sectors
          State : clean
    Device UUID : f8863d07:23d5d4af:dd1d2f40:e366d96e

  Reshape pos'n : 0
  Delta Devices : 1 (5->6)

    Update Time : Mon Jan 18 00:27:39 2016
       Checksum : bc4a9451 - correct
         Events : 551

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
root@nas:~# mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
           Name : nas:Storage  (local to host nas)
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 11721060352 (11178.07 GiB 12002.37 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=944 sectors
          State : clean
    Device UUID : f38040a7:8e6d340e:ac386daa:80f03141

  Reshape pos'n : 0
  Delta Devices : 1 (5->6)

    Update Time : Mon Jan 18 00:27:39 2016
       Checksum : f743cb24 - correct
         Events : 551

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
root@nas:~# mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
           Name : nas:Storage  (local to host nas)
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 11721060352 (11178.07 GiB 12002.37 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=944 sectors
          State : clean
    Device UUID : d829a9b5:ec6f0b86:06d406be:e508fc22

  Reshape pos'n : 0
  Delta Devices : 1 (5->6)

    Update Time : Mon Jan 18 00:27:39 2016
       Checksum : 72e72e79 - correct
         Events : 551

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
root@nas:~# mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
           Name : nas:Storage  (local to host nas)
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 11721060352 (11178.07 GiB 12002.37 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=944 sectors
          State : clean
    Device UUID : 0fbc03cd:4dfb176e:e5474a86:d7752374

  Reshape pos'n : 0
  Delta Devices : 1 (5->6)

    Update Time : Mon Jan 18 00:27:39 2016
       Checksum : 8bb92893 - correct
         Events : 551

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
root@nas:~# mdadm --examine /dev/sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
           Name : nas:Storage  (local to host nas)
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB)
     Array Size : 11721060352 (11178.07 GiB 12002.37 GB)
  Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=944 sectors
          State : clean
    Device UUID : e89405bf:0fa45530:4d4bd566:3af77917

  Reshape pos'n : 0
  Delta Devices : 1 (5->6)

    Update Time : Mon Jan 18 00:27:39 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c3e22f41 - correct
         Events : 551

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Output of mdadm -D /dev/md127
Code:
/dev/md127:
        Version : 1.2
     Raid Level : raid0
  Total Devices : 5
    Persistence : Superblock is persistent

          State : inactive

  Delta Devices : 1, (-1->0)
      New Level : raid6
     New Layout : left-symmetric
  New Chunksize : 512K

           Name : nas:Storage  (local to host nas)
           UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
         Events : 551

    Number   Major   Minor   RaidDevice

       -       8        0        -        /dev/sda
       -       8       16        -        /dev/sdb
       -       8       32        -        /dev/sdc
       -       8       48        -        /dev/sdd
       -       8       64        -        /dev/sde
When trying to force assemble:
Code:
root@nas:~# mdadm --assemble --force /dev/md127 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: Failed to restore critical section for reshape, sorry.
       Possibly you needed to specify the --backup-file
And with "--invalid-backup"
Code:
root@nas:~# mdadm --assemble --force /dev/md127 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde --invalid-backup
mdadm: failed to RUN_ARRAY /dev/md127: Invalid argument
root@nas:~# mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Mon Aug  4 02:03:59 2014
     Raid Level : raid6
  Used Dev Size : -1
   Raid Devices : 6
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Mon Jan 18 00:27:39 2016
          State : active, degraded, Not Started
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

  Delta Devices : 1, (5->6)

           Name : nas:Storage  (local to host nas)
           UUID : b4812ae8:804e182d:f813a5e1:e9ee6da4
         Events : 551

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       5       8       64        2      active sync   /dev/sde
       3       8       32        3      active sync   /dev/sdc
       4       8       48        4      active sync   /dev/sdd
      10       0        0       10      removed
Thanks very much.
 

Yes please! Thanks very much. I had removed the drive I tried to grow with in the outputs above, I'm not sure if that also had reshape position 0. I'm currently trying r-studio (it's currently running automatic raid parameter detection but it's taking a very long time).

Unfortunately I'm not very familiar with mdadm, especially recovery, so I am very lost, any help you could give would be greatly appreciated.

Thanks again
 

rstudio was able to find the offsets but so far has been unable to detect the array, ETA is currently over 5 years, so unfortunately it looks like I'll have to give up on rstudio. What would you try next?

Thanks!
 
Stephen/S Haran of freedatarecovery.us was able to solve the problem for me! I was sceptical at first but he managed to restore my entire array, I definitely recommend his service if anyone is in a similar situation.