Server - Kubernetes (K8S) 运行 PyTorchJob 的 YAML 配置

欢迎关注我的CSDN:https://spike.blog.csdn.net/

本文地址:https://blog.csdn.net/caroline_wendy/article/details/136499768

PyTorchJob 是 Kubernetes 中的自定义资源,用于在 Kubernetes 上运行 PyTorch 训练任务,这是 Kubeflow 组件的一部分,具有稳定的状态,PyTorchJob 允许像管理 Kubernetes 中的其他内置资源一样创建和管理 PyTorch 作业。要使用 PyTorchJob,需要先安装 PyTorch Operator。默认情况下,PyTorch Operator 会作为控制器部署在 training operator 中。

YAML 配置如下,其中:

  • kindPyTorchJob
  • metadata/name,运行的 Job 名称,不要重名
  • 节点使用 Workerreplicas 重复的节点数量,resources 配置 GPU 数量,即支持2机1卡,或1机2卡
  • command 是运行命令

源码:

yaml 复制代码
apiVersion: "kubeflow.org/v1"
kind: PyTorchJob
metadata:
  name: pytorch-simple-001
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 1
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
          labels:
            file-mount: "true"
            user-mount: "true"
        spec:
#          hostNetwork: false  # New
          containers:
            - name: pytorch
              command:
                - /bin/sh
                - -cl
                - "bash k8s/run_grid0_for_gpu1.sh > nohup.test.log 2>&1"
              image: "harbor.[xxx].com/cryoem:v1.3.1"
              imagePullPolicy: Always
              securityContext:  # New
                privileged: false
                capabilities:
                  add: [ "IPC_LOCK" ]
              resources:
                limits:
                  rdma/hca : 1
                  cpu: 12
                  memory: "100G"
                  nvidia.com/gpu: 2
              workingDir: "workspace/cryoem-project/"
              volumeMounts:
                - name: cache-volume  # change the name to your volume on k8s
                  mountPath: /dev/shm
          nodeSelector:
            gpu.device: "a100"  # support 'a10' or 'a100'
            group: "algo2"
          tolerations:
          - effect: NoSchedule
            key: role
            operator: Equal
            value: "algo2"
          volumes:
           - name: cache-volume  # change the name to your volume on k8s
             emptyDir:
                 medium: Memory
                 sizeLimit: "960G"

查看运行情况:

bash 复制代码
kubectl get pytorchjobs
# kubectl delete pytorchjobs pytorch-simple-001
kubectl get pods
kubectl exec -it -n [your name] pytorch-simple-001-worker-0 bash

运行结果:

bash 复制代码
Thu Mar  7 07:39:13 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A800-SXM...  On   | 00000000:58:00.0 Off |                    0 |
| N/A   52C    P0   259W / 400W |   7833MiB / 81920MiB |     93%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A800-SXM...  On   | 00000000:D0:00.0 Off |                    0 |
| N/A   52C    P0   235W / 400W |  12917MiB / 81920MiB |     93%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
相关推荐
K_i13426 分钟前
Kubernetes流量管理:从Ingress到GatewayAPI演进
云原生·容器·kubernetes
蓝色土耳其love2 小时前
centos 7.9 安装单机版k8s
linux·运维·服务器·kubernetes·centos
七度光阴;3 小时前
Docker入门手册
运维·docker·容器
007php0074 小时前
百度面试题解析:Zookeeper、ArrayList、生产者消费者模型及多线程(二)
java·分布式·zookeeper·云原生·职场和发展·eureka·java-zookeeper
weixin_405023375 小时前
使用docker 安装部署easy-mock
运维·docker·容器
Asuncion0075 小时前
Docker核心揭秘:轻量级虚拟化的革命
服务器·开发语言·docker·云原生
ZLRRLZ5 小时前
【Docker】Docker Image(镜像)
运维·docker·容器
小熊h5 小时前
Kubernetes(K8s) —— 部署(保姆级教程)
云原生·容器·kubernetes
祁同伟.7 小时前
【C++】二叉搜索树(图码详解)
开发语言·数据结构·c++·容器·stl
一个处女座的暖男程序猿7 小时前
若依微服务 nacos的配置文件
微服务·云原生·架构