庸鹊套贡一、raid无缓存
3个文件系统：
---u01: hdd 4T*4 raid10，raid无缓存
---u02: hdd 4T*4 raid5，raid无缓存
---u03: ssd 447G*1
测试结果如下：
./testdd.sh /u01 /u02 /u03 > testdd.log.`date +%Y%m%d%H%M` 2>&1 &
vgraid10_local-lv01 7.3T 100G 7.2T 2% /u01 --- direct写入：38.7 MB/s direct读取：151 MB/s cache写入：328 MB/s cache读取：511 MB/s
vgraid5_local-lv01 11T 88G 11T 1% /u02 --- direct写入：7.4 MB/s direct读取：128 MB/s cache写入：77.6 MB/s cache读取：700 MB/s
vgssd_local-lv01 447G 65G 382G 15% /u03 --- direct写入：147 MB/s direct读取：189 MB/s cache写入：387 MB/s cache读取：512 MB/s
结论：
在无缓存（例如RAID卡禁用缓存、设置为Write Through模式，或使用无缓存的RAID卡）的情况下，针对4块磁盘的配置，RAID 10 的直接写入性能【38.7 MB/s】全面且大幅度地优于 RAID 5【7.4 MB/s】。
二、raid缓存模式
确认以上测试 u01/u02 是无缓存模式：
root@host2 \~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdc 8:32 0 7.3T 0 disk
└─vgraid10_local-lv01 253:3 0 7.3T 0 lvm /u01
sdd 8:48 0 10.9T 0 disk
└─vgraid5_local-lv01 253:4 0 10.9T 0 lvm /u02
sdb 8:16 0 447.1G 0 disk
└─vgssd_local-lv01 253:5 0 447G 0 lvm /u03
执行 arcconf getconfig 1 ld 显示 Logical Device number 0 和 1 的配置信息：
root@host2 \~# arcconf getconfig 1 ld
Controllers found: 1
Logical device information
Logical Device number 0
Logical Device name : vd1
Disk Name : /dev/sdc (Disk0) (Bus: 1, Target: 0, Lun: 0)
Block Size of member drives : 512 Bytes
Array : 0
RAID level : 10
Status of Logical Device : Optimal
Size : 7630830 MB
Stripe-unit size : 256 KB
Full Stripe Size : 512 KB
Interface Type : Serial ATA
Device Type : Data
Boot Type : None
Heads : 255
Sectors Per Track : 32
Cylinders : 65535
Caching : Disabled
Mount Points : Not Mounted
LD Acceleration Method : None
SED Encryption : Disabled
Volume Unique Identifier : 600508B1001CF6173057FB8A85255004
Logical Device segment information
Segment : Availability (SizeMB, Protocol, Type, Connector ID, Location) Serial Number
Group 0, Segment 0 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:2) WQB0BYF0
Group 0, Segment 1 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:4) WQB0B5PV
Group 1, Segment 0 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:3) WQB0B5V7
Group 1, Segment 1 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:5) V302WXYF
Logical Device number 1
Logical Device name : vd2
Disk Name : /dev/sdd (Disk0) (Bus: 1, Target: 0, Lun: 1)
Block Size of member drives : 512 Bytes
Array : 1
RAID level : 5
Status of Logical Device : Optimal
Parity Initialization Status : Completed
Size : 11446245 MB
Stripe-unit size : 256 KB
Full Stripe Size : 768 KB
Interface Type : Serial ATA
Device Type : Data
Boot Type : None
Heads : 255
Sectors Per Track : 32
Cylinders : 65535
Caching : Disabled
Mount Points : Not Mounted
LD Acceleration Method : None
SED Encryption : Disabled
Volume Unique Identifier : 600508B1001CE4C11BEB914107DF0141
Array Physical Device Information
Device ID : Availability (SizeMB, Protocol, Type, Connector ID, Location) Serial Number
Device 14 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:6) V3039ZHF
Device 15 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:7) WQB0BY59
Device 16 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:8) WQB0AW76
Device 17 : Present (3815447MB, SATA, HDD, Connector:CN0, Enclosure:1, Slot:9) VB00EL3F
Command completed successfully.
关键信息是这两行：
Caching : Disabled
在 arcconf 设置 磁盘组的 Caching 为 WB 模式：
arcconf SETCACHE 1 LOGICALDRIVE 0 con
arcconf SETCACHE 1 LOGICALDRIVE 1 con
再次查询，已经强行改成了 缓存模式：
Caching : Enabled
再次测试磁盘的IO性能：
./testdd.sh /u01 /u02 > testdd.log.`date +%Y%m%d%H%M` 2>&1 &
vgraid10_local-lv01 7.3T 100G 7.2T 2% /u01 --- direct写入：38.3 MB/s direct读取：400 MB/s cache写入：303 MB/s cache读取：526 MB/s
vgraid5_local-lv01 11T 88G 11T 1% /u02 --- direct写入：7.1 MB/s direct读取：357 MB/s cache写入：33.7 MB/s cache读取：350 MB/s
结论：
打开RAID卡缓存后，raid5的 direct写入 性能仍然很差【7.1 MB/s】，但是 direct读取 性能有了大幅度提升！【128 MB/s ---> 357 MB/s】读取性能飙升而写入性能依旧拉胯，完全符合理论预期。
最后修改回原来设置：
因为 raid 卡无后备电池保护，存在丢数据风险。
arcconf SETCACHE 1 LOGICALDRIVE 0 coff
arcconf SETCACHE 1 LOGICALDRIVE 1 coff
三、测试脚本
vim testdd.sh
#!/bin/bash
if $# -lt 1 ; then
echo "usage: $0 /target1 /target2 /target3 ..."
exit 1
fi
while $# -gt 0 ; do
target=$1
echo "${target} direct写入："
sync && echo 3 > /proc/sys/vm/drop_caches
time dd if=/dev/zero of=${target}/dd.out bs=8k count=200000 oflag=direct
echo "${target} direct读取："
sync && echo 3 > /proc/sys/vm/drop_caches
time dd if=${target}/dd.out of=/dev/null bs=8k count=200000 iflag=direct
echo "${target} cache写入："
sync && echo 3 > /proc/sys/vm/drop_caches
time dd if=/dev/zero of=${target}/dd.out bs=8k count=200000
echo "${target} cache读取："
sync && echo 3 > /proc/sys/vm/drop_caches
time dd if=${target}/dd.out of=/dev/null bs=8k count=200000
shift
done
chmod +x testdd.sh
./testdd.sh /u01 /u02 > testdd.log.`date +%Y%m%d%H%M` 2>&1 &