Hadoop:yarn的Rust API接口

今天头一次接触了yarn的Rust API接口,在本地搭建了集群,能够得到每个任务的详细信息。

(一)得到所有任务的所有信息命令:

默认是json格式,也可以指定xml的格式,如(curl --compressed -H "Accept: application/xml" -X)

注意:这里的端口号是集群的resourceManager所在的节点

bash 复制代码
[root@node141 hadoop]# curl -X GET "http://node142:8088/ws/v1/cluster/apps"

由于返回的数据过多,这里截个图:

(二)得到指定任务的所有信息命令:

bash 复制代码
[root@node141 hadoop]# curl --compressed -H "Accept: application/json" -X GET "http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004"

或者
curl -X GET http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004

返回结果:

rust 复制代码
{
    "app": {
        "id": "application_1696942734063_0004",
        "user": "root",
        "name": "word count",
        "queue": "default",
        "state": "FINISHED",
        "finalStatus": "SUCCEEDED",
        "progress": 100.0,
        "trackingUI": "History",
        "trackingUrl": "http://node142:8088/proxy/application_1696942734063_0004/",
        "diagnostics": "",
        "clusterId": 1696942734063,
        "applicationType": "MAPREDUCE",
        "applicationTags": "",
        "priority": 0,
        "startedTime": 1696943373204,
        "launchTime": 1696943373820,
        "finishedTime": 1696943387557,
        "elapsedTime": 14353,
        "amContainerLogs": "http://node141:8042/node/containerlogs/container_1696942734063_0004_01_000001/root",
        "amHostHttpAddress": "node141:8042",
        "amRPCAddress": "node141:34142",
        "masterNodeId": "node141:37268",
        "allocatedMB": -1,
        "allocatedVCores": -1,
        "reservedMB": -1,
        "reservedVCores": -1,
        "runningContainers": -1,
        "memorySeconds": 48979,
        "vcoreSeconds": 26,
        "queueUsagePercentage": 0.0,
        "clusterUsagePercentage": 0.0,
        "resourceSecondsMap": {
            "entry": {
                "key": "memory-mb",
                "value": "48979"
            },
            "entry": {
                "key": "vcores",
                "value": "26"
            }
        },
        "preemptedResourceMB": 0,
        "preemptedResourceVCores": 0,
        "numNonAMContainerPreempted": 0,
        "numAMContainerPreempted": 0,
        "preemptedMemorySeconds": 0,
        "preemptedVcoreSeconds": 0,
        "preemptedResourceSecondsMap": {
            
        },
        "logAggregationStatus": "SUCCEEDED",
        "unmanagedApplication": false,
        "amNodeLabelExpression": "",
        "timeouts": {
            "timeout": [
                {
                    "type": "LIFETIME",
                    "expiryTime": "UNLIMITED",
                    "remainingTimeInSeconds": -1
                }
            ]
        }
    }
}

(三)过滤出state == "FINISHED"的json------失败

bash 复制代码
curl http://node142:8088/ws/v1/cluster/apps | jq '.apps.app[] | select(.state == "FINISHED")' > /opt/data/running_apps.json
javascript 复制代码
[root@node141 ~]# cat /opt/data/running_apps.json 
{
  "id": "application_1696942734063_0001",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0002",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0003",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
{
  "id": "application_1696942734063_0004",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
相关推荐
Tim风声(网络工程师)2 小时前
不同射频对应不同mac地址(查找无线用户连接AP信息)
服务器·网络·tcp/ip·智能路由器·无线ap
深空数字孪生4 小时前
储能调峰新实践:智慧能源平台如何保障风电消纳与电网稳定?
大数据·人工智能·物联网
百胜软件@百胜软件4 小时前
胜券POS:打造智能移动终端,让零售智慧运营触手可及
大数据
摩羯座-185690305945 小时前
Python数据可视化基础:使用Matplotlib绘制图表
大数据·python·信息可视化·matplotlib
在未来等你5 小时前
Kafka面试精讲 Day 13:故障检测与自动恢复
大数据·分布式·面试·kafka·消息队列
jiedaodezhuti6 小时前
Flink通讯超时问题深度解析:Akka AskTimeoutException解决方案
大数据·flink
庄小焱6 小时前
大数据存储域——Kafka实战经验总结
大数据·kafka·大数据存储域
鳄鱼杆6 小时前
服务器 | Docker应用开发与部署的实践以及阿里云镜像加速配置
服务器·阿里云·docker
羚羊角uou6 小时前
【Linux】命名管道
linux·运维·服务器
曾经的三心草6 小时前
Python2-工具安装使用-anaconda-jupyter-PyCharm-Matplotlib
android·java·服务器