HadoopYarn常用命令、yarn application查看任务、yarn logs查看日志、yarn applicationattempt查看尝试运行的任务、查看容器、Yarn生产环境核心参数

Redamancy_062023-12-22 19:50

文章目录

  • [16. Yarn资源调度器](#16. Yarn资源调度器)
    • [16.5 Yarn常用命令](#16.5 Yarn常用命令)
      • [16.5.1 yarn application查看任务](#16.5.1 yarn application查看任务)
        • [16.5.1.1 列出所有Application：](#16.5.1.1 列出所有Application：)
        • [16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）](#16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）)
        • [16.5.1.3 Kill掉Application：](#16.5.1.3 Kill掉Application：)
      • [16.5.2 yarn logs查看日志](#16.5.2 yarn logs查看日志)
        • [16.5.2.1 查询Application日志：](#16.5.2.1 查询Application日志：)
        • [16.5.2.2 查询Container日志：](#16.5.2.2 查询Container日志：)
      • [16.5.3 yarn applicationattempt查看尝试运行的任务](#16.5.3 yarn applicationattempt查看尝试运行的任务)
        • [16.5.3.1 列出所有Application尝试的列表：](#16.5.3.1 列出所有Application尝试的列表：)
        • [16.5.3.2 打印ApplicationAttemp状态：](#16.5.3.2 打印ApplicationAttemp状态：)
      • [16.5.4 yarn container查看容器](#16.5.4 yarn container查看容器)
        • [16.5.4.1 列出所有Application 尝试的列表](#16.5.4.1 列出所有Application 尝试的列表)
        • [16.5.4.2 打印Container状态：](#16.5.4.2 打印Container状态：)
      • [16.5.5 yarn node查看节点状态](#16.5.5 yarn node查看节点状态)
      • [16.5.6 yarn rmadmin更新配置](#16.5.6 yarn rmadmin更新配置)
      • [16.5.7 yarn queue查看队列](#16.5.7 yarn queue查看队列)
      • [16.6 Yarn生产环境核心参数](#16.6 Yarn生产环境核心参数)

16. Yarn资源调度器

16.5 Yarn常用命令

Yarn状态的查询，除了可以在hadoop103:8088页面查看外，还可以通过命令操作。常见的命令操作如下所示：

需求：执行WordCount案例，并用Yarn命令查看任务运行情况。

16.5.1 yarn application查看任务

16.5.1.1 列出所有Application：
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-10-10 16:57:24,620 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL

这里因为任务已经执行结束了，所以没有显示出来。

16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-10-10 17:03:48,178 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1665325770064_0003	          word count	           MAPREDUCE	    summer	   default	          FINISHED	         SUCCEEDED	           100%	http://hadoop102:19888/jobhistory/job/job_1665325770064_0003

这里的任务application_1665325770064_0003就是刚刚我们执行的任务

16.5.1.3 Kill掉Application：
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1665325770064_0003
2022-10-10 17:05:56,667 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application application_1665325770064_0003 has already finished

因为application_1665325770064_0003这个任务已经完成了，所以不能再次被kill了

16.5.2 yarn logs查看日志

16.5.2.1 查询Application日志：
powershell 复制代码 
yarn logs -applicationId <ApplicationId>
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003
16.5.2.2 查询Container日志：
powershell 复制代码 
yarn logs -applicationId <ApplicationId> -containerId <ContainerId>
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003 -containerId container_1665325770064_0003_01_000001

containerId从16.5.3.1里获取

16.5.3 yarn applicationattempt查看尝试运行的任务

16.5.3.1 列出所有Application尝试的列表：
powershell 复制代码 
yarn applicationattempt -list <ApplicationId>
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0003
2022-10-10 20:01:36,619 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0003_000001	            FINISHED	container_1665325770064_0003_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0003/
16.5.3.2 打印ApplicationAttemp状态：
powershell 复制代码 
yarn applicationattempt -status <ApplicationAttemptId>
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1665325770064_0003_000001
2022-10-10 20:11:35,446 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application Attempt Report : 
	ApplicationAttempt-Id : appattempt_1665325770064_0003_000001
	State : FINISHED
	AMContainer : container_1665325770064_0003_01_000001
	Tracking-URL : http://hadoop103:8088/proxy/application_1665325770064_0003/
	RPC Port : 42572
	AM Host : hadoop103
	Diagnostics :

ApplicationAttemptId从16.5.3.1里获取

16.5.4 yarn container查看容器

16.5.4.1 列出所有Application 尝试的列表
powershell 复制代码 
yarn container -list <ApplicationAttemptId>
powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1665325770064_0003_000001
2022-10-10 20:15:25,743 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of containers :0
                  Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL

只有在任务跑的途中才能看到 container 的状态，因为这个任务运行完之后，container容器会立即释放，我这里就没有container容器，因为任务已经运行结束了

powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0010
2022-10-10 20:25:08,250 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0010_000001	             RUNNING	container_1665325770064_0010_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0010/

这个靠手速弄出来一个结果，这个就是有容器的时候里面显示的内容

16.5.4.2 打印Container状态：
powershell 复制代码 
yarn container -status <ContainerId>

[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0003_01_000001

2022-10-10 20:20:13,494 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032

Container with id 'container_1665325770064_0003_01_000001' doesn't exist in RM or Timeline Server.

注：只有在任务跑的途中才能看到 container 的状态

powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0010_01_000001
2022-10-10 20:25:14,642 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Container Report : 
	Container-Id : container_1665325770064_0010_01_000001
	Start-Time : 1665404700780
	Finish-Time : 0
	State : RUNNING
	Execution-Type : GUARANTEED
	LOG-URL : http://hadoop104:8042/node/containerlogs/container_1665325770064_0010_01_000001/summer
	Host : hadoop104:36495
	NodeHttpAddress : http://hadoop104:8042
	Diagnostics : null

这个靠手速弄了一个任务在运行中的显示结果

16.5.5 yarn node查看节点状态

列出所有节点：yarn node -list -all

powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn node -list -all
2022-10-10 20:33:40,711 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total Nodes:3
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
 hadoop104:36495	        RUNNING	   hadoop104:8042	                           0
 hadoop103:38375	        RUNNING	   hadoop103:8042	                           0
 hadoop102:40891	        RUNNING	   hadoop102:8042	                           0

16.5.6 yarn rmadmin更新配置

加载队列配置：yarn rmadmin -refreshQueues

powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
2022-10-10 20:39:09,817 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8033

16.5.7 yarn queue查看队列

打印队列信息：yarn queue -status

powershell 复制代码 
[summer@hadoop102 hadoop-3.1.3]$ yarn queue -status default
2022-10-10 20:43:10,934 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Queue Information : 
Queue Name : default
	State : RUNNING
	Capacity : 100.0%
	Current Capacity : .0%
	Maximum Capacity : 100.0%
	Default Node Label expression : <DEFAULT_PARTITION>
	Accessible Node Labels : *
	Preemption : disabled
	Intra-queue Preemption : disabled


这里的值更详细

16.6 Yarn生产环境核心参数

相关推荐
atbigapp.com1 小时前
AI数据分析：一键生成可视化分析思路
大数据·人工智能·数据分析
叶域1 小时前
正则表达式(复习)
大数据·python·正则表达式
Data-Miner2 小时前
电商数据分析 电商平台销售数据分析 电商平台数据库设计 揭秘电商怎么做数据分析
大数据
hellolianhua2 小时前
大数据hadoop课程笔记
大数据·hadoop·笔记
乙卯年QAQ2 小时前
【Hive】Hive安装
数据仓库·hive·hadoop
AAEllisonPang2 小时前
Spark性能优化深度剖析：十大实战策略与案例解析
大数据·性能优化·spark
青云交3 小时前
Java 大视界 -- 基于 Java 的大数据分布式数据库架构设计与实践（125）
java·大数据·分布式·分布式数据库·架构设计·数据处理·高可用性
用户Taobaoapi20144 小时前
淘宝商品评论 API 接口详解
大数据
不爱学习的小枫5 小时前
spark常见的submit参数
大数据·分布式·spark
闯闯桑5 小时前
Hive 中用于小文件合并的配置参数
数据仓库·hive·hadoop
热门推荐
01如何在WPS和Word/Excel中直接使用DeepSeek功能02DeepSeek各版本说明与优缺点分析03本地部署DeepSeek教程（Mac版本）04从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑05如何本地部署AI智能体平台，带你手搓一个AI Agent06DeepSeek R1本地化部署 Ollama + Chatbox 打造最强 AI 工具07DeepSeek RAGFlow构建本地知识库系统08Windows 11 安装 Dify 完整指南 非docker环境09本地化部署AI知识库：基于Ollama+DeepSeek+AnythingLLM保姆级教程10网络安全 - DDoS 攻击原理 + 实验