Hadoop:yarn的Rust API接口

今天头一次接触了yarn的Rust API接口,在本地搭建了集群,能够得到每个任务的详细信息。

(一)得到所有任务的所有信息命令:

默认是json格式,也可以指定xml的格式,如(curl --compressed -H "Accept: application/xml" -X)

注意:这里的端口号是集群的resourceManager所在的节点

bash 复制代码
[root@node141 hadoop]# curl -X GET "http://node142:8088/ws/v1/cluster/apps"

由于返回的数据过多,这里截个图:

(二)得到指定任务的所有信息命令:

bash 复制代码
[root@node141 hadoop]# curl --compressed -H "Accept: application/json" -X GET "http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004"

或者
curl -X GET http://node142:8088/ws/v1/cluster/apps/application_1696942734063_0004

返回结果:

rust 复制代码
{
    "app": {
        "id": "application_1696942734063_0004",
        "user": "root",
        "name": "word count",
        "queue": "default",
        "state": "FINISHED",
        "finalStatus": "SUCCEEDED",
        "progress": 100.0,
        "trackingUI": "History",
        "trackingUrl": "http://node142:8088/proxy/application_1696942734063_0004/",
        "diagnostics": "",
        "clusterId": 1696942734063,
        "applicationType": "MAPREDUCE",
        "applicationTags": "",
        "priority": 0,
        "startedTime": 1696943373204,
        "launchTime": 1696943373820,
        "finishedTime": 1696943387557,
        "elapsedTime": 14353,
        "amContainerLogs": "http://node141:8042/node/containerlogs/container_1696942734063_0004_01_000001/root",
        "amHostHttpAddress": "node141:8042",
        "amRPCAddress": "node141:34142",
        "masterNodeId": "node141:37268",
        "allocatedMB": -1,
        "allocatedVCores": -1,
        "reservedMB": -1,
        "reservedVCores": -1,
        "runningContainers": -1,
        "memorySeconds": 48979,
        "vcoreSeconds": 26,
        "queueUsagePercentage": 0.0,
        "clusterUsagePercentage": 0.0,
        "resourceSecondsMap": {
            "entry": {
                "key": "memory-mb",
                "value": "48979"
            },
            "entry": {
                "key": "vcores",
                "value": "26"
            }
        },
        "preemptedResourceMB": 0,
        "preemptedResourceVCores": 0,
        "numNonAMContainerPreempted": 0,
        "numAMContainerPreempted": 0,
        "preemptedMemorySeconds": 0,
        "preemptedVcoreSeconds": 0,
        "preemptedResourceSecondsMap": {
            
        },
        "logAggregationStatus": "SUCCEEDED",
        "unmanagedApplication": false,
        "amNodeLabelExpression": "",
        "timeouts": {
            "timeout": [
                {
                    "type": "LIFETIME",
                    "expiryTime": "UNLIMITED",
                    "remainingTimeInSeconds": -1
                }
            ]
        }
    }
}

(三)过滤出state == "FINISHED"的json------失败

bash 复制代码
curl http://node142:8088/ws/v1/cluster/apps | jq '.apps.app[] | select(.state == "FINISHED")' > /opt/data/running_apps.json
javascript 复制代码
[root@node141 ~]# cat /opt/data/running_apps.json 
{
  "id": "application_1696942734063_0001",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0002",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
 ......
}
{
  "id": "application_1696942734063_0003",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
{
  "id": "application_1696942734063_0004",
  "user": "root",
  "name": "word count",
  "queue": "default",
  "state": "FINISHED",
  ......
}
相关推荐
阿里云大数据AI技术12 分钟前
ODPS 15周年开发者活动|征文+动手实践双赛道开启,参与活动赢定制好礼!
大数据·人工智能·云计算
19H14 分钟前
Flink-Source算子点位提交问题(Earliest)
大数据·flink
阿巴~阿巴~2 小时前
Linux 第一个系统程序 - 进度条
linux·服务器·bash
运器1232 小时前
【一起来学AI大模型】支持向量机(SVM):核心算法深度解析
大数据·人工智能·算法·机器学习·支持向量机·ai·ai编程
我科绝伦(Huanhuan Zhou)3 小时前
华为泰山服务器重启后出现 XFS 文件系统磁盘“不识别”(无法挂载或访问),但挂载点目录仍在且无数据
运维·服务器·华为
望获linux3 小时前
【Linux基础知识系列】第四十三篇 - 基础正则表达式与 grep/sed
linux·运维·服务器·开发语言·前端·操作系统·嵌入式软件
万米商云4 小时前
企业物资集采平台解决方案:跨地域、多仓库、百部门——大型企业如何用一套系统管好百万级物资?
大数据·运维·人工智能
BigData共享4 小时前
极致性能背后的黑科技?这个世上没有“银弹”!(三)
大数据
阿里云大数据AI技术4 小时前
Flink Forward Asia 2025 主旨演讲精彩回顾
大数据·人工智能·flink
y_y_liang4 小时前
图生生AI商品换背景,高效商拍!
大数据·人工智能·ai·ai作画