Server - Kubernetes (K8S) 运行 PyTorchJob 的 YAML 配置

欢迎关注我的CSDN:https://spike.blog.csdn.net/

本文地址:https://blog.csdn.net/caroline_wendy/article/details/136499768

PyTorchJob 是 Kubernetes 中的自定义资源,用于在 Kubernetes 上运行 PyTorch 训练任务,这是 Kubeflow 组件的一部分,具有稳定的状态,PyTorchJob 允许像管理 Kubernetes 中的其他内置资源一样创建和管理 PyTorch 作业。要使用 PyTorchJob,需要先安装 PyTorch Operator。默认情况下,PyTorch Operator 会作为控制器部署在 training operator 中。

YAML 配置如下,其中:

  • kindPyTorchJob
  • metadata/name,运行的 Job 名称,不要重名
  • 节点使用 Workerreplicas 重复的节点数量,resources 配置 GPU 数量,即支持2机1卡,或1机2卡
  • command 是运行命令

源码:

yaml 复制代码
apiVersion: "kubeflow.org/v1"
kind: PyTorchJob
metadata:
  name: pytorch-simple-001
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 1
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
          labels:
            file-mount: "true"
            user-mount: "true"
        spec:
#          hostNetwork: false  # New
          containers:
            - name: pytorch
              command:
                - /bin/sh
                - -cl
                - "bash k8s/run_grid0_for_gpu1.sh > nohup.test.log 2>&1"
              image: "harbor.[xxx].com/cryoem:v1.3.1"
              imagePullPolicy: Always
              securityContext:  # New
                privileged: false
                capabilities:
                  add: [ "IPC_LOCK" ]
              resources:
                limits:
                  rdma/hca : 1
                  cpu: 12
                  memory: "100G"
                  nvidia.com/gpu: 2
              workingDir: "workspace/cryoem-project/"
              volumeMounts:
                - name: cache-volume  # change the name to your volume on k8s
                  mountPath: /dev/shm
          nodeSelector:
            gpu.device: "a100"  # support 'a10' or 'a100'
            group: "algo2"
          tolerations:
          - effect: NoSchedule
            key: role
            operator: Equal
            value: "algo2"
          volumes:
           - name: cache-volume  # change the name to your volume on k8s
             emptyDir:
                 medium: Memory
                 sizeLimit: "960G"

查看运行情况:

bash 复制代码
kubectl get pytorchjobs
# kubectl delete pytorchjobs pytorch-simple-001
kubectl get pods
kubectl exec -it -n [your name] pytorch-simple-001-worker-0 bash

运行结果:

bash 复制代码
Thu Mar  7 07:39:13 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A800-SXM...  On   | 00000000:58:00.0 Off |                    0 |
| N/A   52C    P0   259W / 400W |   7833MiB / 81920MiB |     93%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A800-SXM...  On   | 00000000:D0:00.0 Off |                    0 |
| N/A   52C    P0   235W / 400W |  12917MiB / 81920MiB |     93%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
相关推荐
IT成长日记1 小时前
【Docker基础】Docker数据持久化与卷(Volume)介绍
运维·docker·容器·数据持久化·volume·
疯子的模样6 小时前
Docker 安装 Neo4j 保姆级教程
docker·容器·neo4j
虚伪的空想家6 小时前
rook-ceph配置dashboard代理无法访问
ceph·云原生·k8s·存储·rook
庸子9 小时前
基于Jenkins和Kubernetes构建DevOps自动化运维管理平台
运维·kubernetes·jenkins
Lpy25699 小时前
Docker Desktop 安装到D盘(包括镜像下载等)+ 汉化
运维·docker·容器
好奇的菜鸟10 小时前
Docker 配置项详解与示例
运维·docker·容器
Connie145112 小时前
k8s多集群管理中的联邦和舰队如何理解?
云原生·容器·kubernetes
IT成长日记16 小时前
【Docker基础】Docker数据卷管理:docker volume inspect及其参数详解
运维·docker·容器·volume·inspect
伤不起bb16 小时前
Kubernetes 服务发布基础
云原生·容器·kubernetes
ladymorgana16 小时前
【Docker】如何设置 `wiredTigerCacheSizeGB` 和 `resources.limits.memory`
运维·docker·容器