HadoopYarn常用命令、yarn application查看任务、yarn logs查看日志、yarn applicationattempt查看尝试运行的任务、查看容器、Yarn生产环境核心参数

文章目录

• [16. Yarn资源调度器](#16. Yarn资源调度器)
• • [16.5 Yarn常用命令](#16.5 Yarn常用命令)
  • • [16.5.1 yarn application查看任务](#16.5.1 yarn application查看任务)
    • • [16.5.1.1 列出所有Application：](#16.5.1.1 列出所有Application：)
      • [16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）](#16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）)
      • [16.5.1.3 Kill掉Application：](#16.5.1.3 Kill掉Application：)
    • [16.5.2 yarn logs查看日志](#16.5.2 yarn logs查看日志)
    • • [16.5.2.1 查询Application日志：](#16.5.2.1 查询Application日志：)
      • [16.5.2.2 查询Container日志：](#16.5.2.2 查询Container日志：)
    • [16.5.3 yarn applicationattempt查看尝试运行的任务](#16.5.3 yarn applicationattempt查看尝试运行的任务)
    • • [16.5.3.1 列出所有Application尝试的列表：](#16.5.3.1 列出所有Application尝试的列表：)
      • [16.5.3.2 打印ApplicationAttemp状态：](#16.5.3.2 打印ApplicationAttemp状态：)
    • [16.5.4 yarn container查看容器](#16.5.4 yarn container查看容器)
    • • [16.5.4.1 列出所有Application 尝试的列表](#16.5.4.1 列出所有Application 尝试的列表)
      • [16.5.4.2 打印Container状态：](#16.5.4.2 打印Container状态：)
    • [16.5.5 yarn node查看节点状态](#16.5.5 yarn node查看节点状态)
    • [16.5.6 yarn rmadmin更新配置](#16.5.6 yarn rmadmin更新配置)
    • [16.5.7 yarn queue查看队列](#16.5.7 yarn queue查看队列)
    • [16.6 Yarn生产环境核心参数](#16.6 Yarn生产环境核心参数)

16. Yarn资源调度器

16.5 Yarn常用命令

Yarn状态的查询，除了可以在hadoop103:8088页面查看外，还可以通过命令操作。常见的命令操作如下所示：

需求：执行WordCount案例，并用Yarn命令查看任务运行情况。

• [summer@hadoop102 ~]$ myhadoop.sh start
  这个是之前写的脚本，想了解的可用看我之前写的文章
  https://blog.csdn.net/Redamancy06/article/details/126234179

• [summer@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /testinput /testoutput/output1
  

16.5.1 yarn application查看任务

16.5.1.1 列出所有Application：

[summer@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-10-10 16:57:24,620 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL

这里因为任务已经执行结束了，所以没有显示出来。

16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）

[summer@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-10-10 17:03:48,178 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1665325770064_0003	          word count	           MAPREDUCE	    summer	   default	          FINISHED	         SUCCEEDED	           100%	http://hadoop102:19888/jobhistory/job/job_1665325770064_0003

这里的任务application_1665325770064_0003就是刚刚我们执行的任务

16.5.1.3 Kill掉Application：

[summer@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1665325770064_0003
2022-10-10 17:05:56,667 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application application_1665325770064_0003 has already finished 

因为application_1665325770064_0003这个任务已经完成了，所以不能再次被kill了

16.5.2 yarn logs查看日志

16.5.2.1 查询Application日志：

yarn logs -applicationId <ApplicationId>

[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003

16.5.2.2 查询Container日志：

yarn logs -applicationId <ApplicationId> -containerId <ContainerId> 

[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003 -containerId container_1665325770064_0003_01_000001

containerId从16.5.3.1里获取

16.5.3 yarn applicationattempt查看尝试运行的任务

16.5.3.1 列出所有Application尝试的列表：

yarn applicationattempt -list <ApplicationId>

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0003
2022-10-10 20:01:36,619 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0003_000001	            FINISHED	container_1665325770064_0003_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0003/

16.5.3.2 打印ApplicationAttemp状态：

yarn applicationattempt -status <ApplicationAttemptId>

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1665325770064_0003_000001
2022-10-10 20:11:35,446 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application Attempt Report : 
	ApplicationAttempt-Id : appattempt_1665325770064_0003_000001
	State : FINISHED
	AMContainer : container_1665325770064_0003_01_000001
	Tracking-URL : http://hadoop103:8088/proxy/application_1665325770064_0003/
	RPC Port : 42572
	AM Host : hadoop103
	Diagnostics : 

ApplicationAttemptId从16.5.3.1里获取

16.5.4 yarn container查看容器

16.5.4.1 列出所有Application 尝试的列表

yarn container -list <ApplicationAttemptId>

[summer@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1665325770064_0003_000001
2022-10-10 20:15:25,743 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of containers :0
                  Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL

只有在任务跑的途中才能看到 container 的状态，因为这个任务运行完之后，container容器会立即释放，我这里就没有container容器，因为任务已经运行结束了

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0010
2022-10-10 20:25:08,250 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0010_000001	             RUNNING	container_1665325770064_0010_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0010/

这个靠手速弄出来一个结果，这个就是有容器的时候里面显示的内容

16.5.4.2 打印Container状态：

yarn container -status <ContainerId>

[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0003_01_000001

2022-10-10 20:20:13,494 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032

Container with id 'container_1665325770064_0003_01_000001' doesn't exist in RM or Timeline Server.

注：只有在任务跑的途中才能看到 container 的状态

[summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0010_01_000001
2022-10-10 20:25:14,642 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Container Report : 
	Container-Id : container_1665325770064_0010_01_000001
	Start-Time : 1665404700780
	Finish-Time : 0
	State : RUNNING
	Execution-Type : GUARANTEED
	LOG-URL : http://hadoop104:8042/node/containerlogs/container_1665325770064_0010_01_000001/summer
	Host : hadoop104:36495
	NodeHttpAddress : http://hadoop104:8042
	Diagnostics : null

这个靠手速弄了一个任务在运行中的显示结果

16.5.5 yarn node查看节点状态

列出所有节点：yarn node -list -all

[summer@hadoop102 hadoop-3.1.3]$ yarn node -list -all
2022-10-10 20:33:40,711 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total Nodes:3
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
 hadoop104:36495	        RUNNING	   hadoop104:8042	                           0
 hadoop103:38375	        RUNNING	   hadoop103:8042	                           0
 hadoop102:40891	        RUNNING	   hadoop102:8042	                           0

16.5.6 yarn rmadmin更新配置

加载队列配置：yarn rmadmin -refreshQueues

[summer@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
2022-10-10 20:39:09,817 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8033

16.5.7 yarn queue查看队列

打印队列信息：yarn queue -status

[summer@hadoop102 hadoop-3.1.3]$ yarn queue -status default
2022-10-10 20:43:10,934 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Queue Information : 
Queue Name : default
	State : RUNNING
	Capacity : 100.0%
	Current Capacity : .0%
	Maximum Capacity : 100.0%
	Default Node Label expression : <DEFAULT_PARTITION>
	Accessible Node Labels : *
	Preemption : disabled
	Intra-queue Preemption : disabled

这里的值更详细

16.6 Yarn生产环境核心参数