14. HDFS基准测试

当Hadoop集群搭建完成后，为了弄清楚HDFS的读写性能，我们可以对集群做一次基准测试。Hadoop安装包中自带了对HDFS进行基准测试的工具类。

一. 测试写入速度

1. 查看命令帮助

   [root@hadoop1 www]# yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar  TestDFSIO
   2024-12-28 03:59:58,695 INFO fs.TestDFSIO: TestDFSIO.1.8
   Missing arguments.
   Usage: TestDFSIO [genericOptions] -read [-random | -backward | -skip [-skipSize Size]] | -write | -append | -truncate | -clean [-compression codecClassName] [-nrFiles N] [-size Size[B|KB|MB|GB|TB]] [-resFile resultFileName] [-bufferSize Bytes] [-storagePolicy storagePolicyName] [-erasureCodePolicy erasureCodePolicyName]
   
   关键参数:
   -read: 读文件
   -write: 写文件
   -nrFiles: 文件数量
   -size: 文件大小

2. 提交测试写入速度的命令

   # 写入10个文件,每个文件的大小为1G
   yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -write -nrFiles 10 -size 1GB

3. 运行结果

   2024-12-28 04:18:57,771 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:             Date & time: Sat Dec 28 04:18:57 UTC 2024
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:         Number of files: 10
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:  Total MBytes processed: 10240
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:       Throughput mb/sec: 22.33
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:  Average IO rate mb/sec: 22.73
   2024-12-28 04:18:57,771 INFO fs.TestDFSIO:   IO rate std deviation: 3.21
   2024-12-28 04:18:57,772 INFO fs.TestDFSIO:      Test exec time sec: 77.77
   2024-12-28 04:18:57,772 INFO fs.TestDFSIO:

二. 测试读取速度

1. 提交测试读取速度的命令

   # 读取10个文件,每个文件的大小为1G
   yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -read -nrFiles 10 -size 1GB

2. 运行结果

   2024-12-28 04:21:15,709 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:             Date & time: Sat Dec 28 04:21:15 UTC 2024
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:         Number of files: 10
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:  Total MBytes processed: 10240
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:       Throughput mb/sec: 81.52
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:  Average IO rate mb/sec: 99.66
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:   IO rate std deviation: 47.22
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:      Test exec time sec: 43
   2024-12-28 04:21:15,710 INFO fs.TestDFSIO:

3. 清理测试生成的文件

   yarn jar hadoop-3.2.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.4-tests.jar TestDFSIO -clean