4块磁盘做raid5 或 raid10,无缓存直接写入性能哪种方式更好?为此做了一个简单的测试对比,事实胜于雄辩,我们用实际测试数据来得出结论。
一、raid无缓存
3个文件系统:
---u01: hdd 4T*4 raid10,raid无缓存
---u02: hdd 4T*4 raid5,raid无缓存
---u03: ssd 447G*1
测试结果如下:
./testdd.sh /u01 /u02 /u03 > testdd.log.`date +%Y%m%d%H%M` 2>&1 &
vgraid10_local-lv01 7.3T 100G 7.2T 2% /u01 --- direct写入:38.7 MB/s direct读取:151 MB/s cache写入:328 MB/s cache读取:511 MB/s
vgraid5_local-lv01 11T 88G 11T 1% /u02 --- direct写入:7.4 MB/s direct读取:128 MB/s cache写入:77.6 MB/s cache读取:700 MB/s
vgssd_local-lv01 447G 65G 382G 15% /u03 --- direct写入:147 MB/s direct读取:189 MB/s cache写入:387 MB/s cache读取:512 MB/s
结论:
在无缓存(例如RAID卡禁用缓存、设置为Write Through模式,或使用无缓存的RAID卡)的情况下,针对4块磁盘的配置,RAID 10 的直接写入性能【38.7 MB/s】全面且大幅度地优于 RAID 5【7.4 MB/s】。
二、raid缓存模式
确认以上测试 u01/u02 是无缓存模式:
[root@host2 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 0 7.3T 0 disk
└─vgraid10_local-lv01 253:3 0 7.3T 0 lvm /u01
sdd 8:48 0 10.9T 0 disk
└─vgraid5_local-lv01 253:4 0 10.9T 0 lvm /u02
sdb 8:16 0 447.1G 0 disk
└─vgssd_local-lv01 253:5 0 447G 0 lvm /u03
执行 arcconf getconfig 1 ld 显示 Logical Device number 0 和 1 的配置信息:
[root@host2 ~]# arcconf getconfig 1 ld
Controllers found: 1
--------------------------------------------------------
Logical device information
--------------------------------------------------------
Logical Device number 0
Logical Device name : vd1
Disk Name : /dev/sdc (Disk0) (Bus: 1, Target: 0, Lun: 0)
Block Size of member drives : 512 Bytes
Array : 0
RAID level : 10
Status of Logical Device : Optimal
Size : 7630830 MB
Stripe-unit size : 256 KB
Full Stripe Size : 512 KB
Interface Type : Serial ATA
Device Type : Data
Boot Type : None
Heads : 255
Sectors Per Track : 32
Cylinders : 65535
Caching : Disabled
Mount Points : Not Mounted
LD Acceleration Method : None
SED Encryption : Disabled
Volume Unique Identifier : 600508B1001CF6173057FB8A85255004
--------------------------------------------------------
Logical Device segment information
--------------------------------------------------------
Segment : Availability (SizeMB, Protocol, Type, Connector ID, Location) Serial Number
--------------------------------------------------------
Group 0, Segment 0 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:2) WQB0BYF0
Group 0, Segment 1 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:4) WQB0B5PV
Group 1, Segment 0 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:3) WQB0B5V7
Group 1, Segment 1 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:5) V302WXYF
Logical Device number 1
Logical Device name : vd2
Disk Name : /dev/sdd (Disk0) (Bus: 1, Target: 0, Lun: 1)
Block Size of member drives : 512 Bytes
Array : 1
RAID level : 5
Status of Logical Device : Optimal
Parity Initialization Status : Completed
Size : 11446245 MB
Stripe-unit size : 256 KB
Full Stripe Size : 768 KB
Interface Type : Serial ATA
Device Type : Data
Boot Type : None
Heads : 255
Sectors Per Track : 32
Cylinders : 65535
Caching : Disabled
Mount Points : Not Mounted
LD Acceleration Method : None
SED Encryption : Disabled
Volume Unique Identifier : 600508B1001CE4C11BEB914107DF0141
--------------------------------------------------------
Array Physical Device Information
--------------------------------------------------------
Device ID : Availability (SizeMB, Protocol, Type, Connector ID, Location) Serial Number
--------------------------------------------------------
Device 14 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:6) V3039ZHF
Device 15 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:7) WQB0BY59
Device 16 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:8) WQB0AW76
Device 17 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:9) VB00EL3F
Command completed successfully.
关键信息是这两行:
Caching : Disabled
在 arcconf 设置 磁盘组的 Caching 为 WB 模式:
arcconf SETCACHE 1 LOGICALDRIVE 0 con
arcconf SETCACHE 1 LOGICALDRIVE 1 con
再次查询,已经强行改成了 缓存模式:
Caching : Enabled
再次测试磁盘的IO性能:
./testdd.sh /u01 /u02 > testdd.log.`date +%Y%m%d%H%M` 2>&1 &
vgraid10_local-lv01 7.3T 100G 7.2T 2% /u01 --- direct写入:38.3 MB/s direct读取:400 MB/s cache写入:303 MB/s cache读取:526 MB/s
vgraid5_local-lv01 11T 88G 11T 1% /u02 --- direct写入:7.1 MB/s direct读取:357 MB/s cache写入:33.7 MB/s cache读取:350 MB/s
结论:
打开RAID卡缓存后,raid5的 direct写入 性能仍然很差【7.1 MB/s】,但是 direct读取 性能有了大幅度提升!【128 MB/s ---> 357 MB/s】读取性能飙升而写入性能依旧拉胯,完全符合理论预期。
最后修改回原来设置:
因为 raid 卡无后备电池保护,存在丢数据风险。
arcconf SETCACHE 1 LOGICALDRIVE 0 coff
arcconf SETCACHE 1 LOGICALDRIVE 1 coff