Hadoop:yarn的Rust API接口

今天头一次接触了yarn的Rust API接口,在本地搭建了集群,能够得到每个任务的详细信息。

(一)得到所有任务的所有信息命令:

默认是json格式,也可以指定xml的格式,如(curl --compressed -H "Accept: application/xml" -X)

注意:这里的端口号是集群的resourceManager所在的节点

bash 复制代码
[root@node141 hadoop]# curl -X GET "http://node142:8088/ws/v1/cluster/apps"

由于返回的数据过多,这里截个图:

(二)得到指定任务的所有信息命令:

bash 复制代码
[root@node141 hadoop]# curl --compressed -H "Accept: application/json" -X GET "http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004"

或者
curl -X GET http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004

返回结果:

rust 复制代码
{
    "app": {
        "id": "application_1696942734063_0004",
        "user": "root",
        "name": "word count",
        "queue": "default",
        "state": "FINISHED",
        "finalStatus": "SUCCEEDED",
        "progress": 100.0,
        "trackingUI": "History",
        "trackingUrl": "http://node142:8088/proxy/application_1696942734063_0004/",
        "diagnostics": "",
        "clusterId": 1696942734063,
        "applicationType": "MAPREDUCE",
        "applicationTags": "",
        "priority": 0,
        "startedTime": 1696943373204,
        "launchTime": 1696943373820,
        "finishedTime": 1696943387557,
        "elapsedTime": 14353,
        "amContainerLogs": "http://node141:8042/node/containerlogs/container_1696942734063_0004_01_000001/root",
        "amHostHttpAddress": "node141:8042",
        "amRPCAddress": "node141:34142",
        "masterNodeId": "node141:37268",
        "allocatedMB": -1,
        "allocatedVCores": -1,
        "reservedMB": -1,
        "reservedVCores": -1,
        "runningContainers": -1,
        "memorySeconds": 48979,
        "vcoreSeconds": 26,
        "queueUsagePercentage": 0.0,
        "clusterUsagePercentage": 0.0,
        "resourceSecondsMap": {
            "entry": {
                "key": "memory-mb",
                "value": "48979"
            },
            "entry": {
                "key": "vcores",
                "value": "26"
            }
        },
        "preemptedResourceMB": 0,
        "preemptedResourceVCores": 0,
        "numNonAMContainerPreempted": 0,
        "numAMContainerPreempted": 0,
        "preemptedMemorySeconds": 0,
        "preemptedVcoreSeconds": 0,
        "preemptedResourceSecondsMap": {
            
        },
        "logAggregationStatus": "SUCCEEDED",
        "unmanagedApplication": false,
        "amNodeLabelExpression": "",
        "timeouts": {
            "timeout": [
                {
                    "type": "LIFETIME",
                    "expiryTime": "UNLIMITED",
                    "remainingTimeInSeconds": -1
                }
            ]
        }
    }
}

(三)过滤出state == "FINISHED"的json------失败

bash 复制代码
curl http://node142:8088/ws/v1/cluster/apps | jq '.apps.app[] | select(.state == "FINISHED")' > /opt/data/running_apps.json
javascript 复制代码
[root@node141 ~]# cat /opt/data/running_apps.json 
{
  "id": "application_1696942734063_0001",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0002",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0003",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
{
  "id": "application_1696942734063_0004",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
相关推荐
阿里云大数据AI技术4 小时前
大数据公有云市场第一,阿里云占比47%!
大数据
Lx3529 小时前
Hadoop容错机制深度解析:保障作业稳定运行
大数据·hadoop
christine-rr12 小时前
linux常用命令(4)——压缩命令
linux·服务器·redis
東雪蓮☆12 小时前
深入理解 LVS-DR 模式与 Keepalived 高可用集群
linux·运维·服务器·lvs
乌萨奇也要立志学C++13 小时前
【Linux】进程概念(二):进程查看与 fork 初探
linux·运维·服务器
T062051413 小时前
工具变量-5G试点城市DID数据(2014-2025年
大数据
向往鹰的翱翔14 小时前
BKY莱德因:5大黑科技逆转时光
大数据·人工智能·科技·生活·健康医疗
绿箭柠檬茶14 小时前
Ubuntu 服务器配置转发网络访问
服务器·网络·ubuntu
獭.獭.14 小时前
Linux -- 信号【上】
linux·运维·服务器
鸿乃江边鸟14 小时前
向量化和列式存储
大数据·sql·向量化