Upgrading My 1TB Linux Mirror Raid to 2TB, using Mdadm and Parted

A few months ago in another blog post I had a 1TB drive fail in my Ubuntu 12.10 linux mirror array (http://blog.trebacz.com/2012/04/replacing-failing-drive-in-linux-raid-1.html). I decided to replace it with a 2TB drive to have room for further expansion. Using unequal drive sizes allows me to stagger my drive replacements

I just ran out of disk space on the 1 TB partition, so I purchased a new 3TB drive. I really on need 2TB, but the Seagate 3TB drive was just to good of a price to pass up from Amazon. This is the second part of the story about adding a 3TB drive to my existing mirror and the challenges that I ran into.

The whole process is just a few (11) steps

1. I checked the array to make sure everything was fine before starting.

sudo mdadm --detail /dev/md0

/dev/md0:
        Version : 0.90
  Creation Time : Fri Mar 11 22:53:54 2011
     Raid Level : raid1
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Oct 30 19:00:40 2012
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 298307dc:5f5c8273:3b33ca75:284608dc (local to host AMD-ubuntu)
         Events : 0.2061

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1

2. Then run a check on the overall disk usage on the machine:

df -h

Filesystem                 Size  Used Avail Use% Mounted on
/dev/sdb1                  224G   52G  162G  25% /
udev                       1.9G  4.0K  1.9G   1% /dev
tmpfs                      779M  1.5M  777M   1% /run
none                       5.0M     0  5.0M   0% /run/lock
none                       1.9G   21M  1.9G   2% /run/shm
none                       100M     0  100M   0% /run/user
/dev/md0                   917G  813G   96G  90% /media/1TB

3. List out the various disk drives, just in case I run into any unforeseen problems and pick out my target drive to replace:

sudo fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors                       
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63  1953520064   976760001   fd  Linux raid autodetec

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d466c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63   476696744   238348341   83  Linux
/dev/sdb2       476696745   488392064     5847660    5  Extended
/dev/sdb5       476696808   488392064     5847628+  82  Linux swap / Solaris

Disk /dev/md0: 1000.2 GB, 1000202174464 bytes
2 heads, 4 sectors/track, 244189984 cylinders, total 1953519872 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x60be4700

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  3907029167  1953513560   fd  Linux raid autodetect

4. In this case it's the 1TB drive labeled /dev/sda that I want to replace.First thing to do is fail the suspect drive in the md0 array. Checking the status of the raid after this shows the sda1 partition failed (as expected):

sudo mdadm --manage /dev/md0 --fail /dev/sda1

5. Next thing is to remove the drive partition from the md0 array:

sudo mdadm --manage /dev/md0 --remove /dev/sda1

6. Shutdown the machine and physically remove the old 1TB drive and replace it with the new 3TB drive. Reboot the machine.

7. Check to make sure the disk was recognized after the reboot and verify the drive assignment (see that /dev/sda is indeed a 3TB drive now below)

sudo fdisk -l

Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sda doesn't contain a valid partition table

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d466c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63   476696744   238348341   83  Linux
/dev/sdb2       476696745   488392064     5847660    5  Extended
/dev/sdb5       476696808   488392064     5847628+  82  Linux swap / Solaris

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x60be4700

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  3907029167  1953513560   fd  Linux raid autodetect

Disk /dev/md0: 1000.2 GB, 1000202174464 bytes
2 heads, 4 sectors/track, 244189984 cylinders, total 1953519872 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes                                                                          
I/O size (minimum/optimal): 4096 bytes / 4096 bytes                                                                             
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table    

8. Format the new 3TB Drive.

8.1 Tried to format the new 3TB disk drive using fdisk.

sudo fdisk /dev/sda

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x806d8533.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: The size of this disk is 3.0 TB (3000592982016 bytes).
DOS partition table format can not be used on drives for volumes
larger than (2199023255040 bytes) for 512-byte sectors. Use parted(1) and GUID partition table format (GPT).

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Realized that since the new drive is 3TB need to use the parted utility to format the drive. Formatted with fdisk, I'd only have a 2TB partition available.

8.2 Set Partition Table to GPT using Parted mklabel.

We need to create a partition >2TB. So, we should use parted command. Before creating the partition, we should set the disk label to GPT. GPT stands for GUID partition table format (GPT). Used parted’s mklabel command to set disk label to GPT as shown below (typed commands in bold). Print command in parted shows your progress.

sudo parted /dev/sda

GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt                                                      
(parted) print                                                            
Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

8.3 Create >2TB Partition using parted mkpart.

Use parted’s mkpart command as shown below to create partition that is greater than 2TB. In this example, we are creating a partition that is roughly of 3TB in size.

(parted) mkpart primary 0GB 3001GB                                        
(parted) print                                                            
Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  3001GB  3001GB               primary

8.4 Set the new partition to raid type use the set flag command in parted to set the partition to "raid" to use the partition in an mdadm array.

(parted) set 1 raid on                                                    
(parted) print                                                            
Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  3001GB  3001GB               primary  raid

(parted) quit
Information: You may need to update /etc/fstab.                           

9. Add the newly created partition to the existing array

sudo mdadm --manage /dev/md0 --add /dev/sda1

The following command allowed me to monitor the re-sync process that took a few hours. Allow the resync to fully complete before continuing.

sudo watch cat /proc/mdstat

10. Grow the array to the new size.

After allowing the resync to fully complete, you can grow the space on the array to the maximum size by:

sudo mdadm --grow /dev/md0 --size=max

11. Extending the filesystem?

Resize the filesystem to take advantage of the larger array size. Check it when I'm done and we now have a 1.8TB mirrored files system, 1 drive change away from being 3TB's when it's needed.

sudo resize2fs /dev/md0

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 224G 52G 162G 25% /
udev 1.9G 4.0K 1.9G 1% /dev
tmpfs 779M 1.4M 777M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 4.7M 1.9G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/md0 1.8T 811G 1005G 45% /media/1TB

Resources:

http://www.thegeekstuff.com/2012/08/2tb-gtp-parted/ - Great tutorial on using parted to format >2TB drives)
https://raid.wiki.kernel.org/index.php/Growing - Great explanation on how to grow a raid array on Linux
http://blog.trebacz.com/2012/04/replacing-failing-drive-in-linux-raid-1.html - Original arcticle about replacing one drive in a RAID 1 array
http://www.amazon.com/Seagate-Barracuda-3-5-Inch-Internal-ST3000DM001/dp/B005T3GRLY - Seagate 3TB 7200 RPM drive