mdadm raid5 failure. set wrong drive to faulty by accident

noticed a degraded raid where /proc/mdstats marked one device as down.

so i wanted to set the missing drive to faulty and reattach it.

somehow i got the wrong drive, leaving me with an unworking raid.

i tried to re-create the raid with --assume-clean but this did not work. what could i do to recover my data?

as seen below it seems that sdd still contains the superblock with the infos about the degraded raid.

the raid5 contains a cryptsetup luks partition.

[root@archiso ~]# mdadm --examine /dev/sda1 /dev/sda1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : d0e85136:224b9fce:71d459c2:3d4c7c82 Name : archiso:0 (local to host archiso) Creation Time : Thu Sep 9 09:26:17 2010 Raid Level : raid5 Raid Devices : 4

Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB) Array Size : 5860552704 (2794.53 GiB 3000.60 GB) Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : a9525f08:f0c6dd3b:b52f190d:a098df0a

Update Time : Thu Sep  9 09:26:17 2010
   Checksum : 73f26b10 - correct
     Events : 0

     Layout : left-symmetric
 Chunk Size : 512K

Device Role : Active device 0 Array State : AA.A ('A' == active, '.' == missing) [root@archiso ~]# mdadm --examine /dev/sdb1 /dev/sdb1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : d0e85136:224b9fce:71d459c2:3d4c7c82 Name : archiso:0 (local to host archiso) Creation Time : Thu Sep 9 09:26:17 2010 Raid Level : raid5 Raid Devices : 4

Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB) Array Size : 5860552704 (2794.53 GiB 3000.60 GB) Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : eb089cca:c83ab79e:863c8178:eebe1c4f

Update Time : Thu Sep  9 09:26:17 2010
   Checksum : 48adc84c - correct
     Events : 0

     Layout : left-symmetric
 Chunk Size : 512K

Device Role : Active device 1 Array State : AA.A ('A' == active, '.' == missing) [root@archiso ~]# mdadm --examine /dev/sdc1 /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : d0e85136:224b9fce:71d459c2:3d4c7c82 Name : archiso:0 (local to host archiso) Creation Time : Thu Sep 9 09:26:17 2010 Raid Level : raid5 Raid Devices : 4

Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB) Array Size : 5860552704 (2794.53 GiB 3000.60 GB) Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : cd7e5164:da8037a5:8ba219fc:efe5b05c

Update Time : Thu Sep  9 09:26:17 2010
   Checksum : 7a101148 - correct
     Events : 0

     Layout : left-symmetric
 Chunk Size : 512K

Device Role : Active device 3 Array State : AA.A ('A' == active, '.' == missing) [root@archiso ~]# mdadm --examine /dev/sdd1 /dev/sdd1: Magic : a92b4efc Version : 0.90.00 UUID : 748b6c50:c383a4fd:e9ac99f0:185f7f8f Creation Time : Thu Jun 4 18:50:33 2009 Raid Level : raid5 Used Dev Size : 976759936 (931.51 GiB 1000.20 GB) Array Size : 2930279808 (2794.53 GiB 3000.61 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0

Update Time : Thu Sep  9 06:58:49 2010
      State : clean

Active Devices : 2 Working Devices : 4 Failed Devices : 2 Spare Devices : 2 Checksum : b4728202 - correct Events : 2170510

     Layout : left-symmetric
 Chunk Size : 64K

  Number   Major   Minor   RaidDevice State

this 3 8 49 3 active sync /dev/sdd1

0 0 8 1 0 active sync /dev/sda1 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 49 3 active sync /dev/sdd1 4 4 8 33 4 spare /dev/sdc1 5 5 8 17 5 spare /dev/sdb1

the original mdadm.conf line:

ARRAY /dev/md0 level=raid5 num-devices=4 metadata=0.90 UUID=748b6c50:c383a4fd:e9ac99f0:185f7f8f

From serverfault yawniek

• i tried to re-create the raid with --assume-clean but this did not work.

  This is what you should do. What do you mean by "did not work"? What's the message? What happened? Did you called mdadm with ALL of the original RAID array partitions?

  what could i do to recover my data?

  Restore from backup. If you have no backup, that's a well-deserved lesson (this is harsh and not intended to be a joke at all).

  edit : given that this is an encrypted volume, you have absolutely zero chance to restore any data if you can't get the RAID correctly working. Can you post just the /proc/mdstat content? I don't understand what's the current state (your message states 2 failed drives, but only 1 failed drive is represented).

  yawniek : no i did not use all, i added missing instead of sdd1 since that one should be the one which was degraded. the backup joke is old and not relevant to my question. there should be no corrupt data thus the array should be recoverable somehow.

  Bart Silverstrim : It wasn't a joke. If you're a sysadmin, anything that drills the importance of backups isn't a joke. Second, if your system overwrote part of a "good drive", it's corrupt. Unless you can find a way to read the raw data and recover some semblance of a file table or partition table, and hope temporary data or formatting information wasn't written over the bits on the drive, it's a loss. You could try something like testdisk to see if you can recover data to another media. RAID is NOT a backup.

  yawniek : sure. but it would be much less work to bring up the raid if possible than recover from the backup. i just don't like the general assumption that someone who has data problems has no backup and therefore can be laughed at. the backup discussion is absolutely irrelevant to the technical problem.

  JamesRyan : as soon as the drive has degraded, if you can't immediatly reenable it uncorrupted then the data on it can't be trusted and the quickest, least hassle way is to restore from backup. Thats why whenever someone comes up with a question like this they get flack for not using/having backups.

  Bart Silverstrim : There wasn't an assumption that you didn't have backups, he said that if the array has been screwed up and the partition damaged/volume formatted, rather than spend five hours trying to fiddle with settings and hope for the best you spend four hours running your restore routine. You ran commands against the array, something went wrong, and it may have made the situation worse. So if there isn't a "quick fix", the answer is restore from backups.
  From wazoox