This is a text-only version of the following page on https://raymii.org: --- Title : Linux software raid, rebuilding broken raid 1 Author : Remy van Elst Date : 14-04-2014 URL : https://raymii.org/s/blog/Linux_software_raid_rebuilding_broken_raid_1.html Format : Markdown/HTML --- Last week Nagios alerted me about a broken disk in one of my clients testing servers. There is a best effort SLA on the thing, and there were spare drives of the same type and size in the datacenter. Lucky me. This particular data center is on biking distance, so I enjoyed a sunny ride there.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

Simply put, I needed to replace the disk and rebuild the raid 1 array. This server is a simple Ubuntu 12.04 LTS server with two disks running in raid 1, no spare. Client has a tight budget, and with a best effort SLA not in production, fine with me. Consultant tip, make sure you have those things signed. The `_` in the `cat /proc/mdstat` tells me the second disk (`/dev/sdb`) has failed: Personalities : [raid1] [raid6] [raid5] [raid4] md0 : active raid1 sda1[0] sdb1[1] 129596288 blocks [2/2] [U_] `U` means up, `_` means down [[source]][2] First we remove the disk from the RAID array: mdadm --manage /dev/md0 --remove /dev/sdb1 Make sure the server can boot from a degraded RAID array: grep BOOT_DEGRADED /etc/initramfs-tools/conf.d/mdadm If it says true, continue on. If not, add or change it and rebuild the initramfs using the following command: update-initramfs -u (Thank you [Karssen][3]) We can now safely shut down the server: shutdown -h 10 Replacing the disk was an issue on itself, it is a [Supermicro 512L-260B][4] chassis where the disks are not in a drive bay, rather they are screwed in from the bottom. Therefore the whole server needs to be removed from the rack (no rails...) when replacing the disk. Normally I would replace them while the server is on, but this server has no hot swap disks so that would be kind of an issue in a full rack. After that, boot the server from the first disk (via the BIOS/UEFI). Make sure you boot to recovery mode. Select the root shell and mount the disk read/write: mount -o remount,rw /dev/sda1 Now copy the partition table to the new (in my case, empty) disk: sfdisk -d /dev/sda > sfdisk /dev/sdb This will erase data on the new disk. Add the disk to the RAID array and wait for the rebuilding to be complete: mdadm --manage /dev/md0 --add /dev/sdb1 This is a nice progress command: watch cat /proc/mdstat It will take a while on large disks: Personalities : [raid1] [raid6] [raid5] [raid4] md0 : active raid1 sda1[0] sdb1[1] 129596288 blocks [2/2] [U_] [=>...................] recovery = 2.6% (343392/129596288) finish=67min speed=98840K/sec unused devices: [1]: https://www.digitalocean.com/?refcode=7435ae6b8212 [2]: https://raid.wiki.kernel.org/index.php/Mdstat#.2Fproc.2Fmdstat [3]: http://blog.karssen.org/2013/01/04/booting-an-ubuntu-server-with-a-degraded-software-raid-array/ [4]: http://www.supermicro.com/products/chassis/1U/512/SC512L-260.cfm --- License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source. This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it. All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Just to be clear, the information on this website is for meant for educational purposes and you use it at your own risk. I do not take responsibility if you screw something up. Use common sense, do not 'rm -rf /' as root for example. If you have any questions then do not hesitate to contact me. See https://raymii.org/s/static/About.html for details.