【大数据】五、yarn基础

清河__2024-03-23 22:56

Yarn

Yarn 是用来做分布式系统中的资源协调技术

MapReduce 1.x

对于 MapReduce 1.x 的版本上：

由 Client 发起计算请求，Job Tracker 接收请求之后分发给各个TaskTrack进行执行

在这个阶段，资源的管理与请求的计算是集成在 mapreduce 上的，这种架构会导致 mapreduce 的功能过于臃肿，也会衍生出一系列的问题。

而 YARN 的出现及时的对这个问题作出了改变，YARN 就类似于一个操作系统，mapreduce 就类似于运行在 YARN 这个操作系统上的实际程序

YARN 同时也支持 Spark、Flink、Taz 等分布式计算技术，这使得 YARN 进一步被发扬光大

YARN 基础

Yarn 的基础原理就是将资源管理 与 作业调度（监视）功能进行拆分，由 ResourceManager 进行资源管理，由 ApplicationMaster 进行作业调度与监视功能

ResourceManager

NodeManager

NodeManager

...

Client 将作业提交给 ResourceManager 进行资源调度

YARN 基础配置

ResourceManager 会将任务分配给一个个的 NodeManager 每个NodeManager 中都有一个个的Contaniner，这一个个的 Container中就保存着一个个的任务的计算，同时，NodeManager 中也保存着一个个任务的 Application Master、每一个 Container 的计算完成后会汇报给 Application Master 其结束的信息，Application Master 会及时向 Resource Manager 汇报其情况信息。

修改 mapred-site.xml：

在最后添加;

xml 复制代码 
<configuration>
        <!-- 指定Mapreduce 的作业执行时，用yarn进行资源调度 -->
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.env</name>
                <value>HADOOP_MAPRED_HOME=/export/server/hadoop-3.3.6</value>
        </property>
        <property>
                <name>mapreduce.map.env</name>
                <value>HADOOP_MAPRED_HOME=/export/server/hadoop-3.3.6</value>
        </property>
        <property>
                <name>mapreduce.reduce.env</name>
                <value>HADOOP_MAPRED_HOME=/export/server/hadoop-3.3.6</value>
        </property>
</configuration>

修改 yarn-site.xml

xml 复制代码 
<configuration>

<!-- Site specific YARN configuration properties -->
    <!-- 设置ResourceManager -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
    </property>
    <!--配置yarn的shuffle服务-->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

在 hadoop-env.sh 中添加：

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

之后将修改好的这几个文件分发给其他节点

scp mapred-site.xml yarn-site.xml hadoop-env.sh node2:$PWD
scp mapred-site.xml yarn-site.xml hadoop-env.sh node3:$PWD

之后就可以打开 yarn.sh：

start-yarn.sh

之后在对应的 8088 端口就可以找到对应的可视化网页信息了

进行词频统计（wordcount）的测试：

hadoop jar /export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /input /output1

就可以顺利调用对应的东西了。

另外，如果我们希望查看日志信息，还需要再 mapred-site.xml 中继续进行配置：

xml 复制代码 
    <property>
        <name>MapReduce.jobhistory.address</name>
        <value>node1:10020</value>
    </property>
    <property>
        <name>MapReduce.jobhistory.webapp.address</name>
        <value>node1:19888</value>
    </property>

继续在 yarn.xml 中配置：

xml 复制代码 
    <!-- 添加如下配置 -->
    <!-- 是否需要开启⽇志聚合 -->
    <!-- 开启⽇志聚合后，将会将各个Container的⽇志保存在yarn.nodemanager.remote-app-logdir的位置 -->
    <!-- 默认保存在/tmp/logs -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <!-- 历史⽇志在HDFS保存的时间，单位是秒 -->
    <!-- 默认的是-1，表示永久保存 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://node1:19888/jobhistory/logs</value>
    </property>

YARN 的任务提交流程

  1. MapReduce 程序运行 Job 任务，创建出一个 JobCommiter，再由 JobCommiter 进行任务提交等工作。

  2. JobCommiter 会将自己要执行的任务提交给 ResourceManager，申请一个应用ID

  3. 若资源足够，ResourceManager 会分配给 MapReduce 一个应用ID，这个时候，Mapreduce 会将自己的程序上传到 HDFS，再由需要程序的节点下载对应的程序来进行运算

  4. JobCommitter 正式向 ResourceManager 提交作业任务，

  5. ResourceManager 会找到一个负载较小的 NodeManager，指定其完成这个任务

  6. 这个被指定的 NodeManager 会创建一个 Container，并创建一个 AppMaster 用来监控这个任务并调度资源（这个 AppMaster 会从 HDFS 上接收对应的信息（分片信息、任务程序）对应每一个分片都对应一个 MapTask）

  7. 在明确了需要的空间之后，这个 AppMaster 会向 ResourceManager 申请资源，ResourceManager 会根据 NodeManager 上的负载情况为其分配对应的 NodeManager

  8. 对应的 Node Manager 会向 HDFS 上下载对应的程序，进行真正的任务计算，并在执行的时候向 APPMaster 进行汇报，例如在成功的时候向 appMaster 进行报告。

YARN 的命令

查看当前在 yarn 中运行的任务：

yarn top

测试一下执行：

hadoop jar /export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 10 10

使用 yarn application 可以查看运行的任务信息

yarn application -list -appStates ALL		# 查看所有任务信息
yarn application -list -appStates FINISHED / RUNNING		# 查看所有已完成的 / 正在运行的

我们也可以根据查看出来的 APPID 直接杀死进程：

yarn application -kill xxxxxxxxxxx

被杀死的任务会被标记为 KILLED，我们可以使用 yarn application -list -appStates KILLED 来进行查看

YARN 调度器

先进先出调度器

如题，先来的先处理，存在饥饿问题哪怕你只需要一毫秒的运行也需要一直等

容量调度器

会开辟两个空间，其中 80% 的资源会用来像 FIFO 一样进行处理，另有 20% 等待处理其他问题，这样就在一定程度上规避了小型任务饥饿问题，但其存在资源浪费问题，因为可能有 20% 的资源始终没有使用

公平调度器

公平调度器会为每个任务分配相同的资源，当有任务执行结束时，其会将其所占有的资源分配给其他任务

YARN 的队列

YARN 默认使用的是 只有一个队列的容量调度器，其实也就是 FIFO，但这里我们可以进行配置，一般情况下，会创建两个队列，一个用来处理主任务，另一个用来处理小任务

这里需要修改配置文件：

修改 etc/hadoop 中的 capacity-scheduler.xml：

xml 复制代码 
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
	<!-- yarn 允许的提交任务的最大数量 -->
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

    <!-- Application Master 允许占集群的资源比例，0.1 代表 10% -->
  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

    <!-- 队列类型 -->
  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare 
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare 
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

    <!-- 默认只有一个 default 队列，这里再添加一个 small 队列来处理小任务 -->
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,small</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

    <!-- 这里是每个队列占用的资源百分比 -->
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>70</value>
    <description>Default queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.capacity</name>
    <value>30</value>
    <description>Default queue target capacity.</description>
  </property>
	<!-- 用户可以占用的资源的比例 -->
  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
	<!-- 每个队列最多占用整体资源的比例 -->
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
  </property>

    <!-- 队列的状态 RUNNING 表示该队列是启用的状态 -->
  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
    
	<!-- 权限管理 允许哪些用户向队列中提交任务 -->
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>
    

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_application_max_priority</name>
    <value>*</value>
    <description>
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.small.acl_application_max_priority</name>
    <value>*</value>
    <description>
      The ACL of who can submit applications with configured priority.
      For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
    </description>
  </property>

   <property>
     <name>yarn.scheduler.capacity.root.default.maximum-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     </description>
   </property>
   <property>
     <name>yarn.scheduler.capacity.root.small.maximum-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Maximum lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        This will be a hard time limit for all applications in this
        queue. If positive value is configured then any application submitted
        to this queue will be killed after exceeds the configured lifetime.
        User can also specify lifetime per application basis in
        application submission context. But user lifetime will be
        overridden if it exceeds queue maximum lifetime. It is point-in-time
        configuration.
        Note : Configuring too low value will result in killing application
        sooner. This feature is applicable only for leaf queue.
     </description>
   </property>

   <property>
     <name>yarn.scheduler.capacity.root.default.default-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     </description>
   </property>
   <property>
     <name>yarn.scheduler.capacity.root.small.default-application-lifetime
     </name>
     <value>-1</value>
     <description>
        Default lifetime of an application which is submitted to a queue
        in seconds. Any value less than or equal to zero will be considered as
        disabled.
        If the user has not submitted application with lifetime value then this
        value will be taken. It is point-in-time configuration.
        Note : Default lifetime can't exceed maximum lifetime. This feature is
        applicable only for leaf queue.
     </description>
   </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers.
      When setting this parameter, the size of the cluster should be taken into account.
      We use 40 as the default value, which is approximately the number of nodes in one rack.
      Note, if this value is -1, the locality constraint in the container request
      will be ignored, which disables the delay scheduling.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.rack-locality-additional-delay</name>
    <value>-1</value>
    <description>
      Number of additional missed scheduling opportunities over the node-locality-delay
      ones, after which the CapacityScheduler attempts to schedule off-switch containers,
      instead of rack-local ones.
      Example: with node-locality-delay=40 and rack-locality-delay=20, the scheduler will
      attempt rack-local assignments after 40 missed opportunities, and off-switch assignments
      after 40+20=60 missed opportunities.
      When setting this parameter, the size of the cluster should be taken into account.
      We use -1 as the default value, which disables this feature. In this case, the number
      of missed opportunities for assigning off-switch containers is calculated based on
      the number of containers and unique locations specified in the resource request,
      as well as the size of the cluster.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name>
    <value>1</value>
    <description>
      Controls the number of OFF_SWITCH assignments allowed
      during a node's heartbeat. Increasing this value can improve
      scheduling rate for OFF_SWITCH containers. Lower values reduce
      "clumping" of applications on particular nodes. The default is 1.
      Legal values are 1-MAX_INT. This config is refreshable.
    </description>
  </property>


  <property>
    <name>yarn.scheduler.capacity.application.fail-fast</name>
    <value>false</value>
    <description>
      Whether RM should fail during recovery if previous applications'
      queue is no longer valid.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.workflow-priority-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to override application priority.
      The syntax for this list is
      [workflowId]:[full_queue_name]:[priority][,next mapping]*
      where an application submitted (or mapped to) queue "full_queue_name"
      and workflowId "workflowId" (as specified in application submission
      context) will be given priority "priority".
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.workflow-priority-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a priority mapping is present, will it override the value specified
      by the user? This can be used by administrators to give applications a
      priority that is different than the one specified by the user.
      The default is false.
    </description>
  </property>

</configuration>

若我们在提交任务时不指定队列，默认会被提交到 default 队列里，另外我们也可以指定我们将队列提交到哪里：

# 默认：
hadoop jar /export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi 10 10

# 指定队列：
hadoop jar /export/server/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount -Dmapreduce.job.queuename=small /input /output3

另外，如果我们希望修改默认提交到的队列，需要在 mapred-site.xml 文件中添加如下配置：

xml 复制代码 
<property>
    <name>mapreduce.job.queuename</name>
    <value>small</value>
</property>
相关推荐
武子康5 分钟前
大数据-133 - ClickHouse 基础概述 全面了解
java·大数据·分布式·clickhouse·flink·spark
shuxianshrng1 小时前
大气网格化精细化监管监测系统
大数据·服务器·windows·经验分享
aqymnkstkw1 小时前
2024年【电气试验】考试题库及电气试验模拟试题
大数据·c语言·人工智能·嵌入式硬件·安全
环能jvav大师2 小时前
基于R语言的统计分析基础：使用dplyr包进行数据操作
大数据·开发语言·数据分析·r语言
不睡懒觉的橙4 小时前
【医疗大数据】医疗保健领域的大数据管理：采用挑战和影响
大数据·人工智能·信息可视化·人机交互·健康医疗
数分大拿的Statham5 小时前
PostgreSQL中的regexp_split_to_table函数详解，拆分字段为多行
大数据·数据库·postgresql·数据分析·数据清洗
howard20056 小时前
大数据时代：历史、发展与未来
大数据
翔云API6 小时前
人证合一接口：智能化身份认证的最佳选择
大数据·开发语言·node.js·ocr·php
知识分享小能手6 小时前
mysql学习教程，从入门到精通，SQL 删除数据（DELETE 语句）（19）
大数据·开发语言·数据库·sql·学习·mysql·数据开发
学习使我快乐——玉祥8 小时前
es查询语法
大数据·elasticsearch·搜索引擎
热门推荐
01RAG 实践- Ollama+RagFlow 部署本地知识库02组基轨迹建模 GBTM的介绍与实现（Stata 或 R）032024年高教社杯数学建模国赛C题超详细解题思路分析04苍穹外卖面试总结05yolov8实战第五天——yolov8+ffmpg实时视频流检测并进行实时推流——（推流，保姆教学）06【2024数模国赛赛题思路公开】国赛B题思路丨附可运行代码丨无偿自提07软件工程从理论到实践客观题汇总(头歌第一章至第八章)08Coze扣子平台完整体验和实践（附国内和国际版对比）09【2024高教社杯全国大学生数学建模竞赛】B题 生产过程中的决策问题——解题思路 代码 论文1051-2 万字长文，深度解读端到端自动驾驶的挑战和前沿