aws(学习笔记第二十五课) 使用aws batch

sealaugh322025-01-19 9:05

aws(学习笔记第二十五课)

  • 使用aws batch

学习内容：

  • aws batch整体架构
  • aws batch的hands on

1. aws batch整体架构

  1. AWS batch的整体架构
    相对online开发而言，batch也是一个另外的一个开发领域。AWS的batch开发，既可以采用EC2，也可以采用Fargate。
    • 用户通过各种方式将job放入到job queue
    • AWS batch将job queue里面的job取出
    • 之后将job通过Fargate或者EC2进行执行

      接下来，hands on采用Fargate的方式进行练习。
  2. AWS batch的创建步骤
    • 创建compute environment
    • 创建job queue，这时候会绑定建立好的compute environment
    • 创建job definition
    • 最后创建job

2.aws batch的hands on

  1. 首先定义计算环境compute environment

    • 定义计算环境

    • 使用Fargate和默认的role

      • 这里的role，使用默认的，之后查看AWS设定该role的权限和信任关系。

      • 查看AWSServiceRoleForBatch

      • 权限设定
        可以看出都是EC2和ECS以及autoscaling相关的权限。

        json 复制代码 
        {
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "AWSBatchPolicyStatement1",
           "Effect": "Allow",
           "Action": [
               "ec2:DescribeAccountAttributes",
               "ec2:DescribeInstances",
               "ec2:DescribeInstanceStatus",
               "ec2:DescribeInstanceAttribute",
               "ec2:DescribeSubnets",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeKeyPairs",
               "ec2:DescribeImages",
               "ec2:DescribeImageAttribute",
               "ec2:DescribeSpotInstanceRequests",
               "ec2:DescribeSpotFleetInstances",
               "ec2:DescribeSpotFleetRequests",
               "ec2:DescribeSpotPriceHistory",
               "ec2:DescribeSpotFleetRequestHistory",
               "ec2:DescribeVpcClassicLink",
               "ec2:DescribeLaunchTemplateVersions",
               "ec2:RequestSpotFleet",
               "autoscaling:DescribeAccountLimits",
               "autoscaling:DescribeAutoScalingGroups",
               "autoscaling:DescribeLaunchConfigurations",
               "autoscaling:DescribeAutoScalingInstances",
               "autoscaling:DescribeScalingActivities",
               "eks:DescribeCluster",
               "ecs:DescribeClusters",
               "ecs:DescribeContainerInstances",
               "ecs:DescribeTaskDefinition",
               "ecs:DescribeTasks",
               "ecs:ListClusters",
               "ecs:ListContainerInstances",
               "ecs:ListTaskDefinitionFamilies",
               "ecs:ListTaskDefinitions",
               "ecs:ListTasks",
               "ecs:DeregisterTaskDefinition",
               "ecs:TagResource",
               "ecs:ListAccountSettings",
               "logs:DescribeLogGroups",
               "iam:GetInstanceProfile",
               "iam:GetRole"
           ],
           "Resource": "*"
       },
       {
           "Sid": "AWSBatchPolicyStatement2",
           "Effect": "Allow",
           "Action": [
               "logs:CreateLogGroup",
               "logs:CreateLogStream"
           ],
           "Resource": "arn:aws:logs:*:*:log-group:/aws/batch/job*"
       },
       {
           "Sid": "AWSBatchPolicyStatement3",
           "Effect": "Allow",
           "Action": [
               "logs:PutLogEvents"
           ],
           "Resource": "arn:aws:logs:*:*:log-group:/aws/batch/job*:log-stream:*"
       },
       {
           "Sid": "AWSBatchPolicyStatement4",
           "Effect": "Allow",
           "Action": [
               "autoscaling:CreateOrUpdateTags"
           ],
           "Resource": "*",
           "Condition": {
               "Null": {
                   "aws:RequestTag/AWSBatchServiceTag": "false"
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement5",
           "Effect": "Allow",
           "Action": "iam:PassRole",
           "Resource": [
               "*"
           ],
           "Condition": {
               "StringEquals": {
                   "iam:PassedToService": [
                       "ec2.amazonaws.com",
                       "ec2.amazonaws.com.cn",
                       "ecs-tasks.amazonaws.com"
                   ]
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement6",
           "Effect": "Allow",
           "Action": "iam:CreateServiceLinkedRole",
           "Resource": "*",
           "Condition": {
               "StringEquals": {
                   "iam:AWSServiceName": [
                       "spot.amazonaws.com",
                       "spotfleet.amazonaws.com",
                       "autoscaling.amazonaws.com",
                       "ecs.amazonaws.com"
                   ]
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement7",
           "Effect": "Allow",
           "Action": [
               "ec2:CreateLaunchTemplate"
           ],
           "Resource": "*",
           "Condition": {
               "Null": {
                   "aws:RequestTag/AWSBatchServiceTag": "false"
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement8",
           "Effect": "Allow",
           "Action": [
               "ec2:TerminateInstances",
               "ec2:CancelSpotFleetRequests",
               "ec2:ModifySpotFleetRequest",
               "ec2:DeleteLaunchTemplate"
           ],
           "Resource": "*",
           "Condition": {
               "Null": {
                   "aws:ResourceTag/AWSBatchServiceTag": "false"
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement9",
           "Effect": "Allow",
           "Action": [
               "autoscaling:CreateLaunchConfiguration",
               "autoscaling:DeleteLaunchConfiguration"
           ],
           "Resource": "arn:aws:autoscaling:*:*:launchConfiguration:*:launchConfigurationName/AWSBatch*"
       },
       {
           "Sid": "AWSBatchPolicyStatement10",
           "Effect": "Allow",
           "Action": [
               "autoscaling:CreateAutoScalingGroup",
               "autoscaling:UpdateAutoScalingGroup",
               "autoscaling:SetDesiredCapacity",
               "autoscaling:DeleteAutoScalingGroup",
               "autoscaling:SuspendProcesses",
               "autoscaling:PutNotificationConfiguration",
               "autoscaling:TerminateInstanceInAutoScalingGroup"
           ],
           "Resource": "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/AWSBatch*"
       },
       {
           "Sid": "AWSBatchPolicyStatement11",
           "Effect": "Allow",
           "Action": [
               "ecs:DeleteCluster",
               "ecs:DeregisterContainerInstance",
               "ecs:RunTask",
               "ecs:StartTask",
               "ecs:StopTask"
           ],
           "Resource": "arn:aws:ecs:*:*:cluster/AWSBatch*"
       },
       {
           "Sid": "AWSBatchPolicyStatement12",
           "Effect": "Allow",
           "Action": [
               "ecs:RunTask",
               "ecs:StartTask",
               "ecs:StopTask"
           ],
           "Resource": "arn:aws:ecs:*:*:task-definition/*"
       },
       {
           "Sid": "AWSBatchPolicyStatement13",
           "Effect": "Allow",
           "Action": [
               "ecs:StopTask"
           ],
           "Resource": "arn:aws:ecs:*:*:task/*/*"
       },
       {
           "Sid": "AWSBatchPolicyStatement14",
           "Effect": "Allow",
           "Action": [
               "ecs:CreateCluster",
               "ecs:RegisterTaskDefinition"
           ],
           "Resource": "*",
           "Condition": {
               "Null": {
                   "aws:RequestTag/AWSBatchServiceTag": "false"
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement15",
           "Effect": "Allow",
           "Action": "ec2:RunInstances",
           "Resource": [
               "arn:aws:ec2:*::image/*",
               "arn:aws:ec2:*::snapshot/*",
               "arn:aws:ec2:*:*:subnet/*",
               "arn:aws:ec2:*:*:network-interface/*",
               "arn:aws:ec2:*:*:security-group/*",
               "arn:aws:ec2:*:*:volume/*",
               "arn:aws:ec2:*:*:key-pair/*",
               "arn:aws:ec2:*:*:launch-template/*",
               "arn:aws:ec2:*:*:placement-group/*",
               "arn:aws:ec2:*:*:capacity-reservation/*",
               "arn:aws:ec2:*:*:elastic-gpu/*",
               "arn:aws:elastic-inference:*:*:elastic-inference-accelerator/*",
               "arn:aws:resource-groups:*:*:group/*"
           ]
       },
       {
           "Sid": "AWSBatchPolicyStatement16",
           "Effect": "Allow",
           "Action": "ec2:RunInstances",
           "Resource": "arn:aws:ec2:*:*:instance/*",
           "Condition": {
               "Null": {
                   "aws:RequestTag/AWSBatchServiceTag": "false"
               }
           }
       },
       {
           "Sid": "AWSBatchPolicyStatement17",
           "Effect": "Allow",
           "Action": [
               "ec2:CreateTags"
           ],
           "Resource": [
               "*"
           ],
           "Condition": {
               "StringEquals": {
                   "ec2:CreateAction": [
                       "RunInstances",
                       "CreateLaunchTemplate",
                       "RequestSpotFleet"
                   ]
               }
           }
       }
   ]
}

      • 信任关系设定

        json 复制代码 
        {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "batch.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

    • 设定VPC等网络环境

      • 设定公网Public VPC
        这里为了方便练习，设定了Public VPC，实际的生产环境中，为了安全考虑，应该是在Private VPC中，但是需要设定NAT 。

    • 创建好的compute environment的执行环境

      等待一会，状态就会变成成功。

    • 定义job queue

      定义job queue很简单，这里需要绑定compute environment。

    • 定义job（job definition）
      Fargate的job定义的思想就是docker image + entrance command的设定。

      • job名字的设定
        • 为执行ECS Task设定role
          • 设定Service
          • 设定权限
            这里选择AWS托管的AmazonECSTaskExecutionRolePolicy。
          • 为job definition设定role
          • 为job definition定义docker image和entrance command

    • 执行job（job execution）

      • 提交新作业
      • 关联job queue和job definition
      • 可以将enterance command覆盖job definition的enterance command
      • 提交新作业

    • 执行结果

      • 执行成功
      • 查看log
        可以看出命令行的覆盖已经成功，job上定义的hello,world from job已经打印出来，job definition里面定义的hello,world from job definition没有生效。
        这里，为了测试，使用的是echo命令，但是真实的生产环境应该是镜像的入口执行command
相关推荐
蒸蒸yyyyzwd3 小时前
cpp 选手秋招学习笔记 day33
c++·笔记·求职招聘
彧azz9 小时前
图的存储结构详解：邻接矩阵的原理、实现与应用
开发语言·数据结构·学习·php
平头哥AI9 小时前
Day 22 _ 包装错误别丢链_%w、errors.Is 与 errors.As
android·服务器·学习·golang·go
一阵寒风9 小时前
CAM智能化助力PCB 智能制造-培训体系第六章.项目开发学习
学习·制造
传奇开心果编程9 小时前
【Xilem 0.4 基础语法学与练】第15课：状态管理与 memoize 性能优化
学习·rust·前端框架
具身AGI9 小时前
视频即仿真，物理AI 人类学习路线 的下一步
人工智能·学习
wuyk55511 小时前
从零吃透 MQTT 通信｜第 8 章 FreeRTOS 多任务架构下 MQTT 工程架构，任务拆分、队列解耦、临界区保护
c语言·开发语言·stm32·学习·架构
陈年老古董11 小时前
PyTorch食物图像分类实战：从数据集制作到CNN模型训练全流程详解
pytorch·深度学习·学习·机器学习·分类·cnn
明志数科11 小时前
从300克Ego头环看第一人称数据采集趋势：设备轻量化之后，场景端壁垒在哪
数码相机·学习
笨鸟先飞的橘猫12 小时前
系统设计第十七天决策卡
学习·游戏
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接03GitHub 镜像站点04历年考研数学一、数学二、数学三真题试卷及答案PDF05【一等奖版】2026 国赛 A题 药材的烘干问题06微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08如何新建文件夹？ 电脑新建文件夹的4种方法09ZCode 周末送额度活动开启：3 亿 Token 免费领取10Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费