HadoopYarn常用命令、yarn application查看任务、yarn logs查看日志、yarn applicationattempt查看尝试运行的任务、查看容器、Yarn生产环境核心参数

文章目录

• [16. Yarn资源调度器](#16. Yarn资源调度器)
• • [16.5 Yarn常用命令](#16.5 Yarn常用命令)
  • • [16.5.1 yarn application查看任务](#16.5.1 yarn application查看任务)
    • • [16.5.1.1 列出所有Application：](#16.5.1.1 列出所有Application：)
      • [16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）](#16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）)
      • [16.5.1.3 Kill掉Application：](#16.5.1.3 Kill掉Application：)
    • [16.5.2 yarn logs查看日志](#16.5.2 yarn logs查看日志)
    • • [16.5.2.1 查询Application日志：](#16.5.2.1 查询Application日志：)
      • [16.5.2.2 查询Container日志：](#16.5.2.2 查询Container日志：)
    • [16.5.3 yarn applicationattempt查看尝试运行的任务](#16.5.3 yarn applicationattempt查看尝试运行的任务)
    • • [16.5.3.1 列出所有Application尝试的列表：](#16.5.3.1 列出所有Application尝试的列表：)
      • [16.5.3.2 打印ApplicationAttemp状态：](#16.5.3.2 打印ApplicationAttemp状态：)
    • [16.5.4 yarn container查看容器](#16.5.4 yarn container查看容器)
    • • [16.5.4.1 列出所有Application 尝试的列表](#16.5.4.1 列出所有Application 尝试的列表)
      • [16.5.4.2 打印Container状态：](#16.5.4.2 打印Container状态：)
    • [16.5.5 yarn node查看节点状态](#16.5.5 yarn node查看节点状态)
    • [16.5.6 yarn rmadmin更新配置](#16.5.6 yarn rmadmin更新配置)
    • [16.5.7 yarn queue查看队列](#16.5.7 yarn queue查看队列)
    • [16.6 Yarn生产环境核心参数](#16.6 Yarn生产环境核心参数)

16. Yarn资源调度器

16.5 Yarn常用命令

Yarn状态的查询，除了可以在hadoop103:8088页面查看外，还可以通过命令操作。常见的命令操作如下所示：

需求：执行WordCount案例，并用Yarn命令查看任务运行情况。

• summer@hadoop102 \~\]$ myhadoop.sh start 这个是之前写的脚本，想了解的可用看我之前写的文章

  

16.5.1 yarn application查看任务

16.5.1.1 列出所有Application：

[summer@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-10-10 16:57:24,620 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL

这里因为任务已经执行结束了，所以没有显示出来。

16.5.1.2 根据Application状态过滤：yarn application -list -appStates （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）

[summer@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-10-10 17:03:48,178 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1665325770064_0003	          word count	           MAPREDUCE	    summer	   default	          FINISHED	         SUCCEEDED	           100%	http://hadoop102:19888/jobhistory/job/job_1665325770064_0003

这里的任务application_1665325770064_0003就是刚刚我们执行的任务

16.5.1.3 Kill掉Application：

[summer@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1665325770064_0003
2022-10-10 17:05:56,667 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application application_1665325770064_0003 has already finished 

因为application_1665325770064_0003这个任务已经完成了，所以不能再次被kill了

16.5.2 yarn logs查看日志

16.5.2.1 查询Application日志：

yarn logs -applicationId <ApplicationId>

[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003

16.5.2.2 查询Container日志：

yarn logs -applicationId <ApplicationId> -containerId <ContainerId> 

[summer@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1665325770064_0003 -containerId container_1665325770064_0003_01_000001

containerId从16.5.3.1里获取

16.5.3 yarn applicationattempt查看尝试运行的任务

16.5.3.1 列出所有Application尝试的列表：

yarn applicationattempt -list <ApplicationId>

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0003
2022-10-10 20:01:36,619 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0003_000001	            FINISHED	container_1665325770064_0003_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0003/

16.5.3.2 打印ApplicationAttemp状态：

yarn applicationattempt -status <ApplicationAttemptId>

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1665325770064_0003_000001
2022-10-10 20:11:35,446 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Application Attempt Report : 
	ApplicationAttempt-Id : appattempt_1665325770064_0003_000001
	State : FINISHED
	AMContainer : container_1665325770064_0003_01_000001
	Tracking-URL : http://hadoop103:8088/proxy/application_1665325770064_0003/
	RPC Port : 42572
	AM Host : hadoop103
	Diagnostics : 

ApplicationAttemptId从16.5.3.1里获取

16.5.4 yarn container查看容器

16.5.4.1 列出所有Application 尝试的列表

yarn container -list <ApplicationAttemptId>

[summer@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1665325770064_0003_000001
2022-10-10 20:15:25,743 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of containers :0
                  Container-Id	          Start Time	         Finish Time	               State	                Host	   Node Http Address	                            LOG-URL

只有在任务跑的途中才能看到 container 的状态，因为这个任务运行完之后，container容器会立即释放，我这里就没有container容器，因为任务已经运行结束了

[summer@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1665325770064_0010
2022-10-10 20:25:08,250 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id	               State	                    AM-Container-Id	                       Tracking-URL
appattempt_1665325770064_0010_000001	             RUNNING	container_1665325770064_0010_01_000001	http://hadoop103:8088/proxy/application_1665325770064_0010/

这个靠手速弄出来一个结果，这个就是有容器的时候里面显示的内容

16.5.4.2 打印Container状态：

yarn container -status <ContainerId>

summer@hadoop102 hadoop-3.1.3\]$ yarn container -status container_1665325770064_0003_01_000001 2022-10-10 20:20:13,494 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032 Container with id 'container_1665325770064_0003_01_000001' doesn't exist in RM or Timeline Server. 注：只有在任务跑的途中才能看到 container 的状态 ```powershell [summer@hadoop102 hadoop-3.1.3]$ yarn container -status container_1665325770064_0010_01_000001 2022-10-10 20:25:14,642 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032 Container Report : Container-Id : container_1665325770064_0010_01_000001 Start-Time : 1665404700780 Finish-Time : 0 State : RUNNING Execution-Type : GUARANTEED LOG-URL : http://hadoop104:8042/node/containerlogs/container_1665325770064_0010_01_000001/summer Host : hadoop104:36495 NodeHttpAddress : http://hadoop104:8042 Diagnostics : null ``` 这个靠手速弄了一个任务在运行中的显示结果 #### 16.5.5 yarn node查看节点状态 列出所有节点：yarn node -list -all ```powershell [summer@hadoop102 hadoop-3.1.3]$ yarn node -list -all 2022-10-10 20:33:40,711 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032 Total Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers hadoop104:36495 RUNNING hadoop104:8042 0 hadoop103:38375 RUNNING hadoop103:8042 0 hadoop102:40891 RUNNING hadoop102:8042 0 ```  #### 16.5.6 yarn rmadmin更新配置 加载队列配置：yarn rmadmin -refreshQueues ```powershell [summer@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues 2022-10-10 20:39:09,817 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8033 ```  #### 16.5.7 yarn queue查看队列 打印队列信息：yarn queue -status ```powershell [summer@hadoop102 hadoop-3.1.3]$ yarn queue -status default 2022-10-10 20:43:10,934 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.159.103:8032 Queue Information : Queue Name : default State : RUNNING Capacity : 100.0% Current Capacity : .0% Maximum Capacity : 100.0% Default Node Label expression : Accessible Node Labels : * Preemption : disabled Intra-queue Preemption : disabled ```   这里的值更详细 #### 16.6 Yarn生产环境核心参数