Saturday 14 April 2007

Converting RAID1 to RAID5 with no data loss

When you record and download lots of TV, movies, music, etc. it can chew up the disk space pretty quickly. If you don't keep on top of your DVD burning you will end up running out of disk space, like me.

That's okay, I have two 200GB drives mirrored in a RAID1 array. If I break the mirror and concatenate the drives I could use all 400GB of space available to me... but if one drive died, I would lose everything that was not backed up.

Buying two more drives isn't the answer as I only have one more PATA drive connection available.

Perhaps I could back everything up, buy two bigger drives, install a new RAID1 array and copy everything back over. That means giving up two perfectly servicable 200GB drives.

If only I could add a third drive and convert the RAID1 array to a RAID5 array. Then I would get the full 400GB of space, and still retain the redundancy. Yeah, right...

Then I stumbled across this blog entry in which a guy creates some experimental loopback devices, creates a RAID1 array and then converts it to a RAID5 array with no data loss.

I was intrigued.

The theory says that the RAID5 algorithm, when applied to 2 disks only, ends up looking like a RAID1 array except for the RAID metadata. If you overwrite the RAID1 metadata with the RAID5 metadata, mdadm should recognise the 2 disk RAID5 array and not mess with the contents.

Once the metadata is updated, you can then add a third partition to the array and grow the RAID5 array to utilise it. All that remains is to then resize the filesystem to fill the new space.

The main question is, am I brave enough to try it?

You bet I am!

Of course, everything is caveated with the usual "back everything up before you attempt this procedure" and, like a good boy, I borrowed a 400GB external drive from work and rsync'd all the important stuff across... and, with heart in mouth, followed the procedure...

Boot from a Fedora Core 6 rescue CDROM and get to a command prompt.

You must ensure you have a recent kernel (> 2.6.17) and that you have a recent version of the mdadm software:

# uname -a 
Linux localhost.localdomain 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:54:20 EDT 2006 i686 unknown  
# mdadm --version
mdadm - v2.5.4 - 13 Ocotober 2006

Stop the array:

# mdadm --stop /dev/md0

Overwrite the RAID1 metadata with the RAID5 metadata:

# mdadm --create /dev/md0 --level=5 -n 2 /dev/hda1 /dev/hdb1 
mdadm: /dev/hda1 appears to contain an ext2fs file system
     size=1946592K  mtime=Sat Apr 14 07:18:32 2007 
mdadm: /dev/hda1 appears to be part of a raid array:
     level=1 devices=2 ctime=Sat Sep 17 16:17:45 2005 
mdadm: /dev/hdb1 appears to contain an ext2fs file system
     size=1946592K  mtime=Sat Apr 17 07:18:32 2007 
mdadm: /dev/hdb1 appears to be part of a raid array:
     level=1 devices=2 ctime=Sat Sep 17 16:17:45 2005 
Continue creating array? y 
mdadm: array /dev/md0 started.

At this point the RAID software decided it wanted to rebuild the array. Uh-oh, there goes my data...

I quickly mounted /dev/md0 and had a look... all my data is still intact! Oh well, let the software do it's thing. Who am I to argue?

Add in the third, new partition:

# mdadm --add /dev/md0 /dev/hdd1

So far, so good. Once the rebuild is complete, grow the RAID5 onto the new partition: (NB: use the --backup-file option in case the grow is interrupted. It will allow a safe recovery.)

# mdadm --grow /dev/md0 --raid-disks=3 --backup-file=/mnt/tmp/raid1-5.backup.file 
mdadm: Need to backup 128K of critical section .. 
mdadm: ... critical section passed.

I'm impressed that I've had no problems so far.

The reshaping of the RAID5 from a 2 disk to a 3 disk array takes quite a while (about 6.5 hours for around 200GB of raw data) but the filesystem resize shouldn't take anywhere near as long:

# e2fsck -f /dev/md0  
# resize2fs -p /dev/md0

Apart from the modification of the RAID metadata, the whole operation can be done "online". I chose to do it from single-user/rescue mode as I wanted to make sure there was no data loss. If you're not too bothered then you could leave the whole thing up and running.

Thanks to Neil Brown for his wonderful software RAID drivers, Nathan Gray for his cunning RAID1/RAID5 conversion technique and, of course, everyone who makes Linux the great OS it is.

25 comments:

  1. Thanks for giving such a good method to convert RAID1 TO RAID5. and your method really working.Thanks

    ReplyDelete
  2. It's indeed a very good method. However, if your root filesystem is a RAID-1 array, don't forget that nor Grub, nor LiLo can read data on a RAID-5 array... Therefore, you need to keep a /boot somewhere outside RAID-5 arrays.My 0.02EUR of experience ;)

    ReplyDelete
  3. Golf Swing Tips29 July 2008 at 23:04

    Sweet external hard drive hack. Thanks for having the balls to try it. Now that such a feat has been duplicated, I think I'll give it a whirl. Thanks for the procedure. Wish me luck. Don't worry, I'll backup everything FIRST.

    ReplyDelete
  4. Hi,followed your instructions which have worked so far (currently it's rebuilding the raid set onto all 3 disks, 7423 minutes left. Yikes!)I had to take a few extra steps to get CentOS boot on the raid after changing it from raid 1 to raid 5:1. Don't mount system disk when booting in rescue mode as then it's not possible to stop the raid set (probably due to that I have an lvm on top).2. After converting from raid 1 to raid 5, reboot again but this time do mount the system image.3. Do chroot /mnt/sysimage4. Find the uid of the modified raid set (in my case /dev/md1): mdadm --detail /dev/md15. Edit /etc/mdadm.conf, change level and uid of the changed raid set.6. Create a new initrd (change kernel version according to your config): mkinitrd /boot/initrd-2.6.25.4-10el5.img 2.6.25.4-10el5Done

    ReplyDelete
  5. One more thing, as I run a lvm on top of the raid I also had to do pvresize /dev/md1 before the volume group and volumes could be extended.

    ReplyDelete
  6. good postgood helpthanks

    ReplyDelete
  7. To be very true, I am not much comfortable with all Linux command prompt because it needs to be very well learned skills. I am happy to know a way to backup all that and to have extra storage. I couldn't imagine that it can be this much easy.eural pass

    ReplyDelete
  8. dress up games junkie10 September 2008 at 16:31

    Got mine working too. Was about to get another set of hard drives before stumbling onto this post.

    ReplyDelete
  9. The chain is no stronger than its weakest link

    ReplyDelete
  10. Myrtle Beach Golf22 October 2008 at 05:37

    Awesome external hard drive hack! You are the man! Thanks for the info brother..

    ReplyDelete
  11. houston storage facilities31 October 2008 at 22:07

    great piece, i appreciate the information, let us know if you have any other info to pass along

    ReplyDelete
  12. RAID1 in RAID5 umwandeln « /var/bergercity/9 February 2009 at 04:19

    [...] nur durch einen anderen RAID-Header unterscheiden. Ausprobiert hat das ganze zum Glück auch schon hier jemand. Soweit die Theorie - die Praxis verursachte bei mir doch ein wenig Gänsehaut, daher habe [...]

    ReplyDelete
  13. Optionally You can create RAID5 with 3 devices, one of them missing:mdadm --create /dev/md0 --level 5 --raid-devices 3 /dev/hda1 /dev/hdb1 missingand later add missing drivemdadm /dev/md0 -a /dev/hdc1

    ReplyDelete
  14. I tried this with Openfiler 2.3 and the mdadm --stop /dev/md1 (in my case) gives a device or resource busy error. Anyway to get it to not be open on boot? Openfiler is probing the raid array on boot, and I can't seem to figure out what to take out of the start out so it won't... ideas?

    ReplyDelete
  15. You will have to unmount the filesystem for the device to not be in use. If this is not possible, try booting from a rescue CD/DVD/floppy/USB-stick. You should then be able to load the RAID drivers and access the device(s) without them being in use.

    ReplyDelete
  16. Hardware Raid1 naar Raid5 brengen zonder dataloss - 9lives7 October 2009 at 00:42

    [...] dat het geen mirror maar een raid5 wordt met nog alle data toegankelijk? volgens dit artikel Converting RAID1 to RAID5 with no data loss scott:// zou het mogelijk zijn met een fedora bootcd? __________________ Microsoft Certified Professional [...]

    ReplyDelete
  17. I have a relatively new raid1 using metadata 1.1 and this method didn't work for me. The newly created raid5 array has about 80 blocks less than the raid1 with the same devices. XFS couldn't find the superblock.For my raid1->raid5 method I used the proven rsync and copying the data from the last raid1 device to the new degraded raid5.

    ReplyDelete
  18. tried this, but the filesystem's superblock was corrupt afterwards. I am not sure what I did wrong. I did issue higher numbers for/sys/block/md0/md/stripe_cache_size/proc/sys/dev/raid/speed_limit_min/proc/sys/dev/raid/speed_limit_maxDuring reshape which sped up reshaping from a slow 400k/s to about a decent 12000 k/s.Reshaping still took more than 2 days and a 3*1.5 TB drives. I guess just fromatting, rebuilding the array and than putting back a backup would have been faster, and less risky.

    ReplyDelete
  19. in follow up to post 17: The new drive I added to the array was faulty. After reboot the bios would not recognize it anymore, so the file system corruption is sort of a logical outcome. I replaced the drive and rebuild the array using my backup to restore data which was a 10 hour job. Everything seems fine now.

    ReplyDelete
  20. Ran across your blog when searching for Raid 1 to Raid 5 Google. Just wanted to say thanks for posting this. It worked like a champ! The resize took about 2-1/2 days going from two 1.5tb drives to three 1.5tb drives.

    ReplyDelete
  21. Here are what other people have been saying...http://mindache.net/wp/http://mindache.net/wp/?cat=4http://mindache.net/wp/?m=200802http://reblog.rot13.org/out/atom.php?user=1&itemtag=linuxhttp://sid.rstack.org/blog/rss.php?cat=Librehttps://wiki.xy88.de/w/index.php?title=Links&oldid=1743https://wiki.xy88.de/wiki/Linkshttp://lopezivan.blogspot.com/2008/04/convertir-un-raid-1-un-raid-5-sin.htmlhttp://sid.rstack.org/blog/index.php/205-de-raid-1-en-raid-5-sans-les-mainshttp://www.davelachapelle.ca/category/servers/http://etbe.coker.com.au/2007/11/16/software-vs-hardware-raid/http://scotgate.org/2006/07/03/growing-a-raid5-array-mdadm/http://www.davelachapelle.ca/2008/07/25/converting-raid-1-to-raid-5/http://www.n8gray.org/blog/2006/09/05/stupid-raid-tricks-with-evms-and-mdadm/http://bugs.eisfair.org/view.php?id=164http://friendfeed.com/goofriderhttp://old.nabble.com/Software-RAID-w--mdadm----Need-Partitions--td22213075.htmlhttp://b.hatena.ne.jp/ual808/http://b.hatena.ne.jp/ual808/rsshttp://www.zwei.net/blog/?feed=rss2http://b.hatena.ne.jp/ual808/linux/http://lopezivan.blogspot.com/2008/04/convertir-un-raid-1-un-raid-5-sin.htmlhttp://www.9lives.be/forum/unix-based-operating-systems/676750-hardware-raid1-naar-raid5-brengen-zonder-dataloss.htmlhttp://www.bergercity.de/linux/raid1-in-raid5-umwandeln/http://www.linuxpedia.fr/doku.php/expert/mdadmhttp://www.vanginderachter.be/2009/raid5/http://b.hatena.ne.jp/entry/scott.wallace.sh/2007/04/converting-raid1-to-raid5-with-no-data-loss/

    ReplyDelete
  22. Jeans Accessories5 March 2010 at 01:48

    so useful,thanks.

    ReplyDelete
  23. It's an interesting way to handle the situation for sure. Alternately, you can go from what 'pm' said (February 12, 2009): you could break the RAID-1 mirror by removing one of the disks from it. You could then add two more drives, and build a RAID-5 array in degraded mode using those drives. Once the array is running, you can copy your stuff over from RAID-1 to RAID-5. Finally remove the remaining RAID-1 disk, and replace it with the third RAID-5 disk, and add it to the array.Doing it this way will save you from the process of extending the RAID-5 array, although I don't really know how long the copying of files would take. I imagine it should be faster though.Obviously, you can use the disks removed from RAID-1 array to build the RAID-5 array as they become available (in which case you'll be well adviced to arrange some other method of backup, just in case a disk fails at critical moment, or something else goes wrong).If you plan to replace all of the disks with bigger ones, and retire your old RAID-1 disks, then your RAID-1 array will remain unaltered during the whole process (even though one of the drives is physically removed from the computer, both of them still independently hold all the data), so if something goes wrong, you still -shouldn't- have lost any of your data. Even so, backup is of course always a good idea.

    ReplyDelete
  24. Yes, thanks for adding to the discussion. There's quite a few different ways to do this type of upgrade but I tried to concentrate on the process of converting a RAID1 array - which was a fairly unique process at the time.

    ReplyDelete