通过mdadm命令调用内核MD模块实现软Raid

1.软raid的实现方式：

 软Raid本身是一个软件程序，这个程序是内核模块MD来模拟实现的。由于它的运行是依赖于操作系统的，软件程序和系统出现故障，RAID就有可能出现问题。性能上不及硬件RAID，还会对CPU造成一定的压力。在生产环境中，不建议使用软Raid，使用raid卡或者Raid芯片实现Raid功能。Linux软体RAID设备通过MD(多个设备)设备驱动程序来实现。

2.mdadm命令介绍：

mdadm:模式化的命令

mdadm [mode] <raiddevice> [options] <component-devices>

-A：--assemble    装配模式,组装一个预先存在的数组。

-C：--build    创建模式,创建一个新的数组

-F：--follow,--monitor    监控模式,选择监控模式。

-G: --grow    改变的有源阵列的大小或形状。

-I: --incremental    添加/从适当的数组中删除单个器件/，并可能启动阵列。

创建的软Raid：/dev/md0, /dev/md1, /dev/md2...

-a {yes|no}:是否为新建的raid设备自动创建设备文件/dev/md#

-l #：指定Raid级别；

-n #:指定用于Raid的块设备的个数，不包括备盘；

-c:指定chunks块的大小

-x #:指定用于备盘的块设备个数；

   DEVICE...

-D，--detail :详细信息 显示一个Raid设备的详细信息。

用法示例：

mdadm -D /dev/md0 等同于mdadm -detail /dev/md0

设备文件：

major:主设备号，用于区分设备类别;

minor:此设备号，用于区分同一种类别下不同的具体设备；

一、通过mdadm命令，调用内核MD模块创建Raid0

1.首先，在虚拟机上新添加两块20G的物理硬盘，用来做在不同磁盘上的Raid。

 注意：在同一块磁盘的不同分区上，不能做raid，当磁盘出现故障，Raid就无法使用了。做Raid的时候，要在不同的物理硬盘上。

1).查看添加好的物理硬盘。

[root@Vmware5 ~]# fdisk -l /dev/sdb Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 [root@Vmware5 ~]# fdisk -l /dev/sdc Disk /dev/sdc: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000

2).创建一个5G大小的sdb1新分区，文件分区类型为fd。

[root@Vmware5 ~]# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x19603682. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-2610, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): +5G Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x19603682 Device Boot Start End Blocks Id System /dev/sdb1 1 654 5253223+ fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

3).创建一个大小为5G的sdc1新分区，文件分区类型为fd。

[root@Vmware5 ~]# fdisk /dev/sdc Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x07b84e42. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-2610, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): +5G Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdc: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x07b84e42 Device Boot Start End Blocks Id System /dev/sdc1 1 654 5253223+ fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

注意：创建好分区之后，不要格式化，格式化的是创建好的软Raid,不是它下边的分区。

4).查看下创建好的分区。

[root@Vmware5 ~]# fdisk -l /dev/sd{b,c}1 Disk /dev/sdb1: 5379 MB, 5379300864 bytes 255 heads, 63 sectors/track, 653 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sdc1: 5379 MB, 5379300864 bytes 255 heads, 63 sectors/track, 653 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000000005）

5).分别添加/dev/sdb1、/dev/sdc1到/dev/sdb、/dev/sdc。

[root@Vmware5 ~]# partx -a /dev/sdb1 /dev/sdb [root@Vmware5 ~]# partx -a /dev/sdc1 /dev/sdc [root@Vmware5 ~]# cat /proc/partitions major minor #blocks name 8 0 10485760 sda 8 1 204800 sda1 8 2 7920640 sda2 8 3 2097152 sda3 8 4 1 sda4 8 5 261120 sda5 8 16 20971520 sdb 8 17 5253223 sdb1 8 32 20971520 sdc 8 33 5253223 sdc1

6).看下有没有其他设备，md0是否被使用。

[root@Vmware5 ~]# cat /proc/mdstat Personalities : unused devices: <none>

7).创建软Raid,名字为md0。

[root@Vmware5 ~]# mdadm -C /dev/md0 -a yes -l 0 -n 2 /dev/sdb1 /dev/sdc1 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.

8).看下是否有了md0这个软Raid设备。

[root@Vmware5 ~]# cat /proc/mdstat Personalities : [raid0] md0 : active raid0 sdc1[1] sdb1[0] 10506240 blocks super 1.2 512k chunks unused devices: <none>

9).软Raid0的详细信息。

[root@Vmware5 ~]# ls /dev/md md-device-map [root@Vmware5 ~]# cat /dev/md/md-device-map md0 1.2 e002e742:690d9a6f:f5fe3e08:bdeac269 /dev/md0

10).在上文提到过，分好区不要急着格式化/dev/sdb1和/dev/sdc1,此时需格式化/dev/md0就行了。

[root@Vmware5 ~]# mke2fs -t ext4 /dev/md0 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=128 blocks, Stripe width=256 blocks 657072 inodes, 2626560 blocks 131328 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2692743168 81 block groups 32768 blocks per group, 32768 fragments per group 8112 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 22 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.

11).创建一个挂在目录，名字为tt1，然后把格式化好的md0分区挂载到tt1上去。

[root@Vmware5 ~]# mkdir -p /tt1 [root@Vmware5 ~]# mount /dev/md0 /tt1 [root@Vmware5 ~]# mount /dev/sda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) /dev/sda3 on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/md0 on /tt1 type ext4 (rw) [root@Vmware5 ~]# cd /tt1/ [root@Vmware5 tt1]# ls lost+found

12).监控md0的详细信息。

[root@Vmware5 tt1]# mdadm -D /dev/md0 /dev/md0: Version : 1.2 版本 Creation Time : Fri Oct 25 05:59:07 2013 创建时间 Raid Level : raid0 Raid类型 Array Size : 10506240 (10.02 GiB 10.76 GB) raid大小 Raid Devices : 2 ##一共有几个设备创建的Raid Total Devices : 2 ##一共有多少设备 Persistence : Superblock is persistent ##超级快是否是持久的 Update Time : Fri Oct 25 05:59:07 2013 ##更新时间 State : clean ##数据是完整地 Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K ##Chunk的大小 Name : Vmware5:0 (local to host Vmware5) 当前主机名 UUID : 42e702e0:6f9a0d69:083efef5:69c2eabd ##设备的UUID Events : 0 Number Major Minor RaidDevice State ##标识号码，主设备号，次设备号 以及磁盘状态 0 8 17 0 active sync /dev/sdb1 1 8 33 1 active sync /dev/sdc1

13).拷贝到tt1目录一下文件，看下是否可以正常查看。

[root@Vmware5 ~]# cp /etc/fstab /tt1/ [root@Vmware5 ~]# ls /tt1 fstab lost+found [root@Vmware5 ~]# cat /tt1/fstab # # /etc/fstab # Created by anaconda on Thu Oct 24 23:49:23 2013 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=b49ee2b3-75aa-4227-a9ff-5d0d990af0fd / ext4 defaults 1 1 UUID=3a69daa4-b393-4694-abbb-b856345b376d /boot ext4 defaults 1 2 UUID=34f85ed8-5f68-4fdc-8aa0-e50d2f9f012e /home ext4 defaults 1 2 UUID=95d97c70-9291-499b-ac16-a38508a85e4d swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0

可以正常查看，说明软Raid0做成功了。

二、通过mdadm命令，创建一个大小为10G的软Raid1。

1).创建一个大小为10G的sdb2新分区，文件分区类型为fd。

[root@Vmware5 ~]# fdisk /dev/sdb WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (655-2610, default 655): Using default value 655 Last cylinder, +cylinders or +size{K,M,G} (655-2610, default 2610): +5G Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): fd Changed system type of partition 2 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x35c1f03a Device Boot Start End Blocks Id System /dev/sdb1 1 654 5253223+ fd Linux raid autodetect /dev/sdb2 655 1308 5253255 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks.

2).创建一个大小为10G的sdc2新分区，文件分区类型为fd。

[root@Vmware5 ~]# fdisk /dev/sdc WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (655-2610, default 655): Using default value 655 Last cylinder, +cylinders or +size{K,M,G} (655-2610, default 2610): +10G Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): fd Changed system type of partition 2 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdc: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x87e3e792 Device Boot Start End Blocks Id System /dev/sdc1 1 654 5253223+ fd Linux raid autodetect /dev/sdc2 655 1960 10490445 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks.

3).分别添加/dev/sdb2、/dev/sdc2到/dev/sdb、/dev/sdc。

[root@Vmware5 ~]# partx -a /dev/sdb2 /dev/sdb [root@Vmware5 ~]# partx -a /dev/sdc2 /dev/sdc

4).用mdadm命令，创建软Raid1，名字为md1。

[root@Vmware5 ~]# mdadm -C /dev/md1 -a yes -l 1 -n 2 /dev/sd{b,c}2 mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: largest drive (/dev/sdc2) exceeds size (5249088K) by more than 1% Continue creating array? yes mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started. [root@Vmware5 ~]# cat /proc/mdstat Personalities : [raid0] [raid1] md1 : active raid1 sdc2[1] sdb2[0] 5249088 blocks super 1.2 [2/2] [UU] md127 : active raid0 sdb1[0] sdc1[1] 10506240 blocks super 1.2 512k chunks unused devices: <none>

5).格式化新建的软raid1名字为md1。

[root@Vmware5 ~]# mke2fs -t ext4 /dev/md1 -L MD1 mke2fs 1.41.12 (17-May-2010) Filesystem label=MD1 OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 328656 inodes, 1312272 blocks 65613 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=1346371584 41 block groups 32768 blocks per group, 32768 fragments per group 8016 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 20 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.

6).用mount命令，临时挂载到/mnt目录下。

[root@Vmware5 ~]# mount /dev/md1 /mnt/ [root@Vmware5 ~]# mount /dev/sda2 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) /dev/sda3 on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/md1 on /mnt type ext4 (rw)

7).看下md1的监控信息。

[root@Vmware5 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Oct 25 09:08:45 2013 Raid Level : raid1 Array Size : 5249088 (5.01 GiB 5.38 GB) Used Dev Size : 5249088 (5.01 GiB 5.38 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Oct 25 09:13:10 2013 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : Vmware5:1 (local to host Vmware5) UUID : 91d7c869:e5b6aa2e:7515e116:dac07875 Events : 17 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 34 1 active sync /dev/sdc2

8).模拟软Raid1中/dev/sdb2这块硬盘损坏。

[root@Vmware5 ~]# mdadm /dev/md1 -f /dev/sdb2 mdadm: set /dev/sdb2 faulty in /dev/md1 [root@Vmware5 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Oct 25 09:08:45 2013 Raid Level : raid1 Array Size : 5249088 (5.01 GiB 5.38 GB) Used Dev Size : 5249088 (5.01 GiB 5.38 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Oct 25 09:14:38 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Name : Vmware5:1 (local to host Vmware5) UUID : 91d7c869:e5b6aa2e:7515e116:dac07875 Events : 18 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 34 1 active sync /dev/sdc2 0 8 18 - faulty spare /dev/sdb2

9).把/dev/sdb2这块硬盘移除。

[root@Vmware5 ~]# mdadm /dev/md1 -r /dev/sdb2 mdadm: hot removed /dev/sdb2 from /dev/md1 [root@Vmware5 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Oct 25 09:08:45 2013 Raid Level : raid1 Array Size : 5249088 (5.01 GiB 5.38 GB) Used Dev Size : 5249088 (5.01 GiB 5.38 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Update Time : Fri Oct 25 09:17:14 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : Vmware5:1 (local to host Vmware5) UUID : 91d7c869:e5b6aa2e:7515e116:dac07875 Events : 33 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 34 1 active sync /dev/sdc2

此时，sdb2这块磁盘显示removed。

10).重建一下sdb2这块磁盘。

[root@Vmware5 ~]# mdadm /dev/md1 -a /dev/sdb2 mdadm: added /dev/sdb2 [root@Vmware5 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Oct 25 09:08:45 2013 Raid Level : raid1 Array Size : 5249088 (5.01 GiB 5.38 GB) Used Dev Size : 5249088 (5.01 GiB 5.38 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Oct 25 09:18:05 2013 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 8% complete Name : Vmware5:1 (local to host Vmware5) UUID : 91d7c869:e5b6aa2e:7515e116:dac07875 Events : 40 Number Major Minor RaidDevice State 2 8 18 0 spare rebuilding /dev/sdb2 1 8 34 1 active sync /dev/sdc2

此时，/dev/sdb2磁盘显示正在重建中。

如果大家想查看同步的动态效果：

[root@Vmware5 ~]# watch -n 1 'cat /proc/mdstat'

11).此时，再模拟/dev/sdc这硬盘损坏。

[root@Vmware5 ~]# mdadm /dev/md1 -f /dev/sdc2 mdadm: set /dev/sdc2 faulty in /dev/md1 [root@Vmware5 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Oct 25 09:08:45 2013 Raid Level : raid1 Array Size : 5249088 (5.01 GiB 5.38 GB) Used Dev Size : 5249088 (5.01 GiB 5.38 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Oct 25 09:20:38 2013 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Name : Vmware5:1 (local to host Vmware5) UUID : 91d7c869:e5b6aa2e:7515e116:dac07875 Events : 59 Number Major Minor RaidDevice State 2 8 18 0 active sync /dev/sdb2 1 0 0 1 removed 1 8 34 - faulty spare /dev/sdc2

12).md1是软Raid1，两块硬盘其中一块出现问题，不耽误使用。看下可否查看mnt目录下的文件。

[root@Vmware5 ~]# cp /etc/inittab /mnt/ [root@Vmware5 ~]# cat /mnt/inittab # inittab is only used by upstart for the default runlevel. # # ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM. # # System initialization is started by /etc/init/rcS.conf # # Individual runlevels are started by /etc/init/rc.conf # # Ctrl-Alt-Delete is handled by /etc/init/control-alt-delete.conf # # Terminal gettys are handled by /etc/init/tty.conf and /etc/init/serial.conf, # with configuration in /etc/sysconfig/init. # # For information on how to write upstart event handlers, or how # upstart works, see init(5), init(8), and initctl(8). # # Default runlevel. The runlevels used are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # id:3:initdefault:

可以正常查看，说明了Raid1创建成功了。

到此为止，实验结。