Hadoop大数据应用：HDFS 集群节点缩容

字节全栈_kYu2025-02-03 18:33

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:16:33 CST 2024

Name: 192.168.204.51:50010 (node01)

Hostname: node01

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 180224 (176 KB)

Non DFS Used: 6029176832 (5.62 GB)

DFS Remaining: 199502282752 (185.80 GB)

DFS Used%: 0.00%

DFS Remaining%: 97.07%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:16:34 CST 2024

Name: 192.168.204.52:50010 (node02)

Hostname: node02

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 286720 (280 KB)

Non DFS Used: 6029291520 (5.62 GB)

DFS Remaining: 199502061568 (185.80 GB)

DFS Used%: 0.00%

DFS Remaining%: 97.07%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:16:34 CST 2024

复制代码 
![](https://img-blog.csdnimg.cn/direct/cdba3285a6634f6f940214a165f58473.png)


（2）查看

root@hadoop \~# cd /usr/local/hadoop/

root@hadoop hadoop# ./bin/hadoop fs -ls /

复制代码 
![](https://img-blog.csdnimg.cn/direct/829f39a9158f4c0aa4a02776b1fe7dae.png)


（3）上传第一个测试文件

root@hadoop \~# /usr/local/hadoop/bin/hadoop fs -put /root/jenkins.tar.gz /jenkins.tar.gz

复制代码 
![](https://img-blog.csdnimg.cn/direct/37c4056e5ac1456ba33e224217725f26.png)


观察上传变化

root@hadoop hadoop# ./bin/hadoop fs -ls /

复制代码 
![](https://img-blog.csdnimg.cn/direct/7a54afa7eab54190ab6eaf9441486e60.png)



（4）上传第二个测试文件

root@hadoop \~# /usr/local/hadoop/bin/hadoop fs -put /root/pinpoint-image.tar.gz /pinpoint.tar.gz

复制代码 
![](https://img-blog.csdnimg.cn/direct/8e36ea6c0f174200b87ed2d1e906a2ff.png)


 观察上传变化

root@hadoop hadoop# ./bin/hadoop fs -ls /

复制代码 
![](https://img-blog.csdnimg.cn/direct/9b630eb381d84f51b3473a359abdc648.png)


（5）web查看

http://192.168.204.50:50070/

复制代码 
![](https://img-blog.csdnimg.cn/direct/d4fd2d03b0ce467890c731838bd9230e.png)



（6）清空本地文件

root@hadoop \~# rm -rf jenkins.tar.gz pinpoint-image.tar.gz

复制代码 
![](https://img-blog.csdnimg.cn/direct/80e5de1590bf4a68a44004626b5a7af8.png)



（7）再次查看节点状态 （已使用）


node01: 2.41GB 


node02: 2.27GB 


node03: 2.15GB  


node04:  2.26GB

root@hadoop hadoop# ./bin/hdfs dfsadmin -report

Configured Capacity: 822126559232 (765.67 GB)

Present Capacity: 798784053248 (743.93 GB)

DFS Remaining: 789019389952 (734.83 GB)

DFS Used: 9764663296 (9.09 GB)

DFS Used%: 1.22%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (4):

Name: 192.168.204.54:50010 (node04)

Hostname: node04

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2424492032 (2.26 GB)

Non DFS Used: 5662638080 (5.27 GB)

DFS Remaining: 197444509696 (183.88 GB)

DFS Used%: 1.18%

DFS Remaining%: 96.07%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:57:57 CST 2024

Name: 192.168.204.53:50010 (node03)

Hostname: node03

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2312085504 (2.15 GB)

Non DFS Used: 5620936704 (5.23 GB)

DFS Remaining: 197598617600 (184.03 GB)

DFS Used%: 1.12%

DFS Remaining%: 96.14%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:57:58 CST 2024

Name: 192.168.204.51:50010 (node01)

Hostname: node01

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2592927744 (2.41 GB)

Non DFS Used: 6029148160 (5.62 GB)

DFS Remaining: 196909563904 (183.39 GB)

DFS Used%: 1.26%

DFS Remaining%: 95.80%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:57:57 CST 2024

Name: 192.168.204.52:50010 (node02)

Hostname: node02

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2435158016 (2.27 GB)

Non DFS Used: 6029783040 (5.62 GB)

DFS Remaining: 197066698752 (183.53 GB)

DFS Used%: 1.18%

DFS Remaining%: 95.88%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 15:57:59 CST 2024

复制代码 
![](https://img-blog.csdnimg.cn/direct/6eb3b8a6c5dd4f2d846d7f918c6b8ad2.png)


（8）计算总存储使用量


9.09 G （jenkins 1.24 GB,pinpoint 3.27 GB ,存储副本量为2）


![](https://img-blog.csdnimg.cn/direct/1221f1c8e6ec475a85e036520546ff2b.png)


(9)修改配置文件（hadoop节点）

#添加代码

root@hadoop hadoop# vim hdfs-site.xml

复制代码 
![](https://img-blog.csdnimg.cn/direct/34b8a8485da742b39d95d3b758e60459.png)


修改前：


![](https://img-blog.csdnimg.cn/direct/7959b026a64c48f0b9e87c3f549d4230.png)


修改后：

......

dfs.hosts.exclude

/usr/local/hadoop/etc/hadoop/exclude

......

复制代码 
![](https://img-blog.csdnimg.cn/direct/24565073220742528932f9881c7a7260.png)




（10）配置移除主机

root@hadoop hadoop# vim /usr/local/hadoop/etc/hadoop/exclude

复制代码 
![](https://img-blog.csdnimg.cn/direct/ff17266e710341d6bd7398462eb58fe5.png)

node04

复制代码 
![](https://img-blog.csdnimg.cn/direct/cba4969f7d634f63b391ea243c42913b.png)



（11）数据刷新

root@hadoop hadoop# ./bin/hdfs dfsadmin -refreshNodes

复制代码 
![](https://img-blog.csdnimg.cn/direct/d69a53fa8a2d46f4917d80357a095745.png)


（12）查看节点状态 


node04节点状态变为Decommissioned in Program

root@hadoop hadoop# ./bin/hdfs dfsadmin -report

Configured Capacity: 619019411456 (576.51 GB)

Present Capacity: 600866701319 (559.60 GB)

DFS Remaining: 590966771712 (550.38 GB)

DFS Used: 9899929607 (9.22 GB)

DFS Used%: 1.65%

Under replicated blocks: 2

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (4):

Name: 192.168.204.54:50010 (node04)

Hostname: node04

Decommission Status : Decommission in progress

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2424492032 (2.26 GB)

Non DFS Used: 5663162368 (5.27 GB)

DFS Remaining: 197443985408 (183.88 GB)

DFS Used%: 1.18%

DFS Remaining%: 96.06%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:34:33 CST 2024

Name: 192.168.204.53:50010 (node03)

Hostname: node03

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2312085504 (2.15 GB)

Non DFS Used: 5756112896 (5.36 GB)

DFS Remaining: 197463441408 (183.90 GB)

DFS Used%: 1.12%

DFS Remaining%: 96.07%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 2

Last contact: Thu Mar 14 16:34:31 CST 2024

Name: 192.168.204.51:50010 (node01)

Hostname: node01

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2592927744 (2.41 GB)

Non DFS Used: 6232051712 (5.80 GB)

DFS Remaining: 196706660352 (183.20 GB)

DFS Used%: 1.26%

DFS Remaining%: 95.71%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 3

Last contact: Thu Mar 14 16:34:33 CST 2024

Name: 192.168.204.52:50010 (node02)

Hostname: node02

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2570424327 (2.39 GB)

Non DFS Used: 6164545529 (5.74 GB)

DFS Remaining: 196796669952 (183.28 GB)

DFS Used%: 1.25%

DFS Remaining%: 95.75%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 2

Last contact: Thu Mar 14 16:34:32 CST 2024

Decommissioning datanodes (1):

Name: 192.168.204.54:50010 (node04)

Hostname: node04

Decommission Status : Decommission in progress

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2424492032 (2.26 GB)

Non DFS Used: 5663162368 (5.27 GB)

DFS Remaining: 197443985408 (183.88 GB)

DFS Used%: 1.18%

DFS Remaining%: 96.06%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:34:33 CST 2024

复制代码 
![](https://img-blog.csdnimg.cn/direct/5d3f672b36484e649c57a298f6482d2a.png)


![](https://img-blog.csdnimg.cn/direct/d41c7d8948d14646a5cca6dfa8edb710.png)


(13) 再次查看节点


node04节点状态变为Decommissioned

root@hadoop hadoop# ./bin/hdfs dfsadmin -report

Configured Capacity: 619019411456 (576.51 GB)

Present Capacity: 601338761216 (560.04 GB)

DFS Remaining: 589149618176 (548.69 GB)

DFS Used: 12189143040 (11.35 GB)

DFS Used%: 2.03%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (4):

Name: 192.168.204.54:50010 (node04)

Hostname: node04

Decommission Status : Decommissioned

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2424492032 (2.26 GB)

Non DFS Used: 5662666752 (5.27 GB)

DFS Remaining: 197444481024 (183.88 GB)

DFS Used%: 1.18%

DFS Remaining%: 96.07%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:42:33 CST 2024

Name: 192.168.204.53:50010 (node03)

Hostname: node03

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 3123707904 (2.91 GB)

Non DFS Used: 5620969472 (5.23 GB)

DFS Remaining: 196786962432 (183.27 GB)

DFS Used%: 1.52%

DFS Remaining%: 95.75%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:42:31 CST 2024

Name: 192.168.204.51:50010 (node01)

Hostname: node01

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2998738944 (2.79 GB)

Non DFS Used: 6029717504 (5.62 GB)

DFS Remaining: 196503183360 (183.01 GB)

DFS Used%: 1.46%

DFS Remaining%: 95.61%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:42:33 CST 2024

Name: 192.168.204.52:50010 (node02)

Hostname: node02

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 3642204160 (3.39 GB)

Non DFS Used: 6029963264 (5.62 GB)

DFS Remaining: 195859472384 (182.41 GB)

DFS Used%: 1.77%

DFS Remaining%: 95.29%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 16:42:32 CST 2024

复制代码 
![](https://img-blog.csdnimg.cn/direct/989b60e886f24f08b20832d3b3468764.png)


## 


(14) 重启

root@hadoop hadoop# ssh node04

root@node04 \~# reboot

复制代码 
![](https://img-blog.csdnimg.cn/direct/3cab152e382f41dcbfb06e034c923ef5.png)


## 


（15）查看节点（已使用）


node01: 2.79GB 


node02: 3.39GB 


node03: 2.91GB

root@hadoop hadoop# ./bin/hdfs dfsadmin -report

Configured Capacity: 619019411456 (576.51 GB)

Present Capacity: 601338118144 (560.04 GB)

DFS Remaining: 589148975104 (548.69 GB)

DFS Used: 12189143040 (11.35 GB)

DFS Used%: 2.03%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

Live datanodes (3):

Name: 192.168.204.53:50010 (node03)

Hostname: node03

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 3123707904 (2.91 GB)

Non DFS Used: 5621551104 (5.24 GB)

DFS Remaining: 196786380800 (183.27 GB)

DFS Used%: 1.52%

DFS Remaining%: 95.75%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 17:00:14 CST 2024

Name: 192.168.204.51:50010 (node01)

Hostname: node01

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 2998738944 (2.79 GB)

Non DFS Used: 6029742080 (5.62 GB)

DFS Remaining: 196503158784 (183.01 GB)

DFS Used%: 1.46%

DFS Remaining%: 95.61%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 17:00:12 CST 2024

Name: 192.168.204.52:50010 (node02)

Hostname: node02

Decommission Status : Normal

Configured Capacity: 205531639808 (191.42 GB)

DFS Used: 3642204160 (3.39 GB)

Non DFS Used: 6030000128 (5.62 GB)

DFS Remaining: 195859435520 (182.41 GB)

DFS Used%: 1.77%

DFS Remaining%: 95.29%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu Mar 14 17:00:12 CST 2024

Dead datanodes (1):

相关推荐
木圭的AI时代指南7 小时前
Spark-X2.5 长输入 Agent 实测：全量读取陷阱与解释器寻找死循环复盘，附可复现任务
大数据·分布式·spark
dongsdh8 小时前
部署python
大数据·elasticsearch·搜索引擎
QYR-分析8 小时前
重轨受电弓行业深度报告：市场格局、技术迭代与发展前景
大数据·数据库·人工智能
镜像视界(浙江)科技有限公司9 小时前
《视频孪生之上：二维展示终结，三维空间计算重构城市逻辑》——跨摄像连续表达 × 三角测量厘米级定位 × 动态轨迹建模，构建新一代城市空间
大数据·人工智能·算法·矩阵·音视频·空间计算
大大大大晴天9 小时前
每天认识一个组件：元数据平台 DataHub
大数据
m0_547486669 小时前
《大数据分析导论》全套PPT课件2026
大数据·大数据分析
starzy199010 小时前
Flink高级之CEP深度剖析：Pattern API、NFA引擎与风控实战
大数据·flink
星禾元亨12 小时前
生成式 AI 时代的架构演进与 GEO 优化：实体企业 AI 落地避坑指南
大数据·人工智能·架构·自动化·创业创新
IT毕设实战小研12 小时前
基于大数据的国内主要农作物产量趋势分析与可视化
android·java·大数据·django·课程设计
OFIRM碳基硅基13 小时前
西游金蝉劫 IP · 之 《金蝉子前传渡缘劫》电影 20 亿票房触发 · 项目专项扶持协议 总览 拉格朗日光影动画-西游金蝉劫项目组
大数据·人工智能·西游金蝉劫·蝎子精·蝎子精女妖王·金蝉子前传渡缘劫
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接03历年考研数学一、数学二、数学三真题试卷及答案PDF04微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新05GitHub 镜像站点06人工智能（AI）发展史：详细里程碑072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08【一等奖版】2026 国赛 A题 药材的烘干问题09ZCode 周末送额度活动开启：3 亿 Token 免费领取10Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费