4. Error Recovery

1. Q: I have a RAID-1 (mirroring) setup, and lost power while there was disk activity. Now what do I do?

   A: The redundancy of RAID levels is designed to protect against a disk failure, not against a power failure. There are several ways to recover from this situation.

   • Method (1): Use the raid tools. These can be used to sync the raid arrays. They do not fix file-system damage; after the raid arrays are sync'ed, then the file-system still has to be fixed with fsck. Raid arrays can be checked with ckraid /etc/raid1.conf (for RAID-1, else, /etc/raid5.conf, etc.) Calling ckraid /etc/raid1.conf --fix will pick one of the disks in the array (usually the first), and use that as the master copy, and copy its blocks to the others in the mirror. To designate which of the disks should be used as the master, you can use the --force-source flag: for example, ckraid /etc/raid1.conf --fix --force-source /dev/hdc3 The ckraid command can be safely run without the --fix option to verify the inactive RAID array without making any changes. When you are comfortable with the proposed changes, supply the --fix option.
   • Method (2): Paranoid, time-consuming, not much better than the first way. Lets assume a two-disk RAID-1 array, consisting of partitions /dev/hda3 and /dev/hdc3. You can try the following:
     1. fsck /dev/hda3
     2. fsck /dev/hdc3
     3. decide which of the two partitions had fewer errors, or were more easily recovered, or recovered the data that you wanted. Pick one, either one, to be your new ``master'' copy. Say you picked /dev/hdc3.
     4. dd if=/dev/hdc3 of=/dev/hda3
     5. mkraid raid1.conf -f --only-superblock
     Instead of the last two steps, you can instead run ckraid /etc/raid1.conf --fix --force-source /dev/hdc3 which should be a bit faster.
   • Method (3): Lazy man's version of above. If you don't want to wait for long fsck's to complete, it is perfectly fine to skip the first three steps above, and move directly to the last two steps. Just be sure to run fsck /dev/md0 after you are done. Method (3) is actually just method (1) in disguise.

   In any case, the above steps will only sync up the raid arrays. The file system probably needs fixing as well: for this, fsck needs to be run on the active, unmounted md device.

   With a three-disk RAID-1 array, there are more possibilities, such as using two disks to ''vote'' a majority answer. Tools to automate this do not currently (September 97) exist.

2. Q: I have a RAID-4 or a RAID-5 (parity) setup, and lost power while there was disk activity. Now what do I do?

   A: The redundancy of RAID levels is designed to protect against a disk failure, not against a power failure. Since the disks in a RAID-4 or RAID-5 array do not contain a file system that fsck can read, there are fewer repair options. You cannot use fsck to do preliminary checking and/or repair; you must use ckraid first.

   The ckraid command can be safely run without the --fix option to verify the inactive RAID array without making any changes. When you are comfortable with the proposed changes, supply the --fix option.

   If you wish, you can try designating one of the disks as a ''failed disk''. Do this with the --suggest-failed-disk-mask flag.

   Only one bit should be set in the flag: RAID-5 cannot recover two failed disks. The mask is a binary bit mask: thus:

       0x1 == first disk
       0x2 == second disk
       0x4 == third disk
       0x8 == fourth disk, etc.
               
   

   Alternately, you can choose to modify the parity sectors, by using the --suggest-fix-parity flag. This will recompute the parity from the other sectors.

   The flags --suggest-failed-dsk-mask and --suggest-fix-parity can be safely used for verification. No changes are made if the --fix flag is not specified. Thus, you can experiment with different possible repair schemes.

3. Q: My RAID-1 device, /dev/md0 consists of two hard drive partitions: /dev/hda3 and /dev/hdc3. Recently, the disk with /dev/hdc3 failed, and was replaced with a new disk. My best friend, who doesn't understand RAID, said that the correct thing to do now is to ''dd if=/dev/hda3 of=/dev/hdc3''. I tried this, but things still don't work.

   A: You should keep your best friend away from you computer. Fortunately, no serious damage has been done. You can recover from this by running:

   mkraid raid1.conf -f --only-superblock
               
   

   By using dd, two identical copies of the partition were created. This is almost correct, except that the RAID-1 kernel extension expects the RAID superblocks to be different. Thus, when you try to reactivate RAID, the software will notice the problem, and deactivate one of the two partitions. By re-creating the superblock, you should have a fully usable system.

4. Q: My version of mkraid doesn't have a --only-superblock flag. What do I do?

   A: The newer tools drop support for this flag, replacing it with the --force-resync flag. It has been reported that the following sequence appears to work with the latest tools and software:

     umount /web (where /dev/md0 was mounted on)
     raidstop /dev/md0
     mkraid /dev/md0 --force-resync --really-force
     raidstart /dev/md0
               
   

   After doing this, a cat /proc/mdstat should report resync in progress, and one should be able to mount /dev/md0 at this point.

5. Q: My RAID-1 device, /dev/md0 consists of two hard drive partitions: /dev/hda3 and /dev/hdc3. My best (girl?)friend, who doesn't understand RAID, ran fsck on /dev/hda3 while I wasn't looking, and now the RAID won't work. What should I do?

   A: You should re-examine your concept of ``best friend''. In general, fsck should never be run on the individual partitions that compose a RAID array. Assuming that neither of the partitions are/were heavily damaged, no data loss has occurred, and the RAID-1 device can be recovered as follows:

   1. make a backup of the file system on /dev/hda3
   2. dd if=/dev/hda3 of=/dev/hdc3
   3. mkraid raid1.conf -f --only-superblock

   This should leave you with a working disk mirror.

6. Q: Why does the above work as a recovery procedure?

   A: Because each of the component partitions in a RAID-1 mirror is a perfectly valid copy of the file system. In a pinch, mirroring can be disabled, and one of the partitions can be mounted and safely run as an ordinary, non-RAID file system. When you are ready to restart using RAID-1, then unmount the partition, and follow the above instructions to restore the mirror. Note that the above works ONLY for RAID-1, and not for any of the other levels.

   It may make you feel more comfortable to reverse the direction of the copy above: copy from the disk that was untouched to the one that was. Just be sure to fsck the final md.

7. Q: I am confused by the above questions, but am not yet bailing out. Is it safe to run fsck /dev/md0 ?

   A: Yes, it is safe to run fsck on the md devices. In fact, this is the only safe place to run fsck.

8. Q: If a disk is slowly failing, will it be obvious which one it is? I am concerned that it won't be, and this confusion could lead to some dangerous decisions by a sysadmin.

   A: Once a disk fails, an error code will be returned from the low level driver to the RAID driver. The RAID driver will mark it as ``bad'' in the RAID superblocks of the ``good'' disks (so we will later know which mirrors are good and which aren't), and continue RAID operation on the remaining operational mirrors.

   This, of course, assumes that the disk and the low level driver can detect a read/write error, and will not silently corrupt data, for example. This is true of current drives (error detection schemes are being used internally), and is the basis of RAID operation.

9. Q: What about hot-repair?

   A: Work is underway to complete ``hot reconstruction''. With this feature, one can add several ``spare'' disks to the RAID set (be it level 1 or 4/5), and once a disk fails, it will be reconstructed on one of the spare disks in run time, without ever needing to shut down the array.

   However, to use this feature, the spare disk must have been declared at boot time, or it must be hot-added, which requires the use of special cabinets and connectors that allow a disk to be added while the electrical power is on.

   As of October 97, there is a beta version of MD that allows:

   • RAID 1 and 5 reconstruction on spare drives
   • RAID-5 parity reconstruction after an unclean shutdown
   • spare disk to be hot-added to an already running RAID 1 or 4/5 array

   By default, automatic reconstruction is (Dec 97) currently disabled by default, due to the preliminary nature of this work. It can be enabled by changing the value of SUPPORT_RECONSTRUCTION in include/linux/md.h.

   If spare drives were configured into the array when it was created and kernel-based reconstruction is enabled, the spare drive will already contain a RAID superblock (written by mkraid), and the kernel will reconstruct its contents automatically (without needing the usual mdstop, replace drive, ckraid, mdrun steps).

   If you are not running automatic reconstruction, and have not configured a hot-spare disk, the procedure described by Gadi Oxman < gadio@netvision.net.il> is recommended:

   • Currently, once the first disk is removed, the RAID set will be running in degraded mode. To restore full operation mode, you need to:
     • stop the array (mdstop /dev/md0)
     • replace the failed drive
     • run ckraid raid.conf to reconstruct its contents
     • run the array again (mdadd, mdrun).
     At this point, the array will be running with all the drives, and again protects against a failure of a single drive.

   Currently, it is not possible to assign single hot-spare disk to several arrays. Each array requires it's own hot-spare.

10. Q: I would like to have an audible alarm for ``you schmuck, one disk in the mirror is down'', so that the novice sysadmin knows that there is a problem.

    A: The kernel is logging the event with a ``KERN_ALERT'' priority in syslog. There are several software packages that will monitor the syslog files, and beep the PC speaker, call a pager, send e-mail, etc. automatically.

11. Q: How do I run RAID-5 in degraded mode (with one disk failed, and not yet replaced)?

    A: Gadi Oxman < gadio@netvision.net.il> writes: Normally, to run a RAID-5 set of n drives you have to:

    mdadd /dev/md0 /dev/disk1 ... /dev/disk(n)
    mdrun -p5 /dev/md0
                  
    

    Even if one of the disks has failed, you still have to mdadd it as you would in a normal setup. (?? try using /dev/null in place of the failed disk ??? watch out) Then, The array will be active in degraded mode with (n - 1) drives. If ``mdrun'' fails, the kernel has noticed an error (for example, several faulty drives, or an unclean shutdown). Use ``dmesg'' to display the kernel error messages from ``mdrun''. If the raid-5 set is corrupted due to a power loss, rather than a disk crash, one can try to recover by creating a new RAID superblock:

    mkraid -f --only-superblock raid5.conf
                
    

    A RAID array doesn't provide protection against a power failure or a kernel crash, and can't guarantee correct recovery. Rebuilding the superblock will simply cause the system to ignore the condition by marking all the drives as ``OK'', as if nothing happened.

12. Q: How does RAID-5 work when a disk fails?

    A: The typical operating scenario is as follows:

    • A RAID-5 array is active.
    • One drive fails while the array is active.
    • The drive firmware and the low-level Linux disk/controller drivers detect the failure and report an error code to the MD driver.
    • The MD driver continues to provide an error-free /dev/md0 device to the higher levels of the kernel (with a performance degradation) by using the remaining operational drives.
    • The sysadmin can umount /dev/md0 and mdstop /dev/md0 as usual.
    • If the failed drive is not replaced, the sysadmin can still start the array in degraded mode as usual, by running mdadd and mdrun.

13. Q:

    A:

14. Q: Why is there no question 13?

    A: If you are concerned about RAID, High Availability, and UPS, then its probably a good idea to be superstitious as well. It can't hurt, can it?

15. Q: I just replaced a failed disk in a RAID-5 array. After rebuilding the array, fsck is reporting many, many errors. Is this normal?

    A: No. And, unless you ran fsck in "verify only; do not update" mode, its quite possible that you have corrupted your data. Unfortunately, a not-uncommon scenario is one of accidentally changing the disk order in a RAID-5 array, after replacing a hard drive. Although the RAID superblock stores the proper order, not all tools use this information. In particular, the current version of ckraid will use the information specified with the -f flag (typically, the file /etc/raid5.conf) instead of the data in the superblock. If the specified order is incorrect, then the replaced disk will be reconstructed incorrectly. The symptom of this kind of mistake seems to be heavy & numerous fsck errors.

    And, in case you are wondering, yes, someone lost all of their data by making this mistake. Making a tape backup of all data before reconfiguring a RAID array is strongly recommended.

16. Q: The QuickStart says that mdstop is just to make sure that the disks are sync'ed. Is this REALLY necessary? Isn't unmounting the file systems enough?

    A: The command mdstop /dev/md0 will:

    • mark it ''clean''. This allows us to detect unclean shutdowns, for example due to a power failure or a kernel crash.
    • sync the array. This is less important after unmounting a filesystem, but is important if the /dev/md0 is accessed directly rather than through a filesystem (for example, by e2fsck).