电商返利APP容器编排实践：K8s在多环境部署中的资源调度优化

大家好，我是省赚客APP研发者阿可！省赚客APP（juwatech.cn）后端由 30+ 微服务组成，涵盖用户中心、订单处理、返利计算、AI推荐等模块，需在开发、测试、预发、生产四套环境中稳定运行。早期采用静态 VM 部署，存在资源浪费、扩缩容滞后、环境不一致等问题。自 2023 年起，我们全面迁移至 Kubernetes，并基于命名空间隔离、HPA 自动扩缩、ResourceQuota 限制及 Pod 拓扑分布策略，实现资源利用率提升 40%、故障恢复时间缩短至 30 秒内。本文结合 YAML 配置与 Java 应用适配代码，详解 K8s 多环境编排的核心实践。

命名空间与环境隔离

每个环境对应独立 Namespace，通过 RBAC 控制访问权限：

yaml 复制代码

# env-prod.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: juwatech-prod
  labels:
    env: production

---
apiVersion: v1
kind: Namespace
metadata:
  name: juwatech-staging
  labels:
    env: staging

CI/CD 流水线（基于 GitLab CI）根据分支自动部署到对应 Namespace：

yaml 复制代码

# .gitlab-ci.yml 片段
deploy-prod:
  stage: deploy
  script:
    - kubectl apply -f k8s/prod/ -n juwatech-prod
  only:
    - main

资源请求与限制精细化配置

Java 应用基于 Spring Boot，启动时指定堆内存，并通过 JVM 参数适配容器环境：

java 复制代码

// juwatech.cn.Application.java
@SpringBootApplication
public class Application {
    public static void main(String[] args) {
        // 确保 JVM 能感知容器内存限制
        System.setProperty("java.awt.headless", "true");
        SpringApplication.run(Application.class, args);
    }
}

对应的 Deployment 设置 requests/limits，避免节点资源争抢：

yaml 复制代码

# deployment-order-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: juwatech-prod
spec:
  replicas: 6
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: registry.juwatech.cn/order-service:1.4.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "512Mi"
            cpu: "200m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        env:
        - name: JAVA_OPTS
          value: "-Xms512m -Xmx768m -XX:+UseG1GC -Dfile.encoding=UTF-8"

HPA 基于自定义指标自动扩缩

除 CPU/Memory 外，我们接入 Prometheus 监控 QPS，并配置 HPA：

yaml 复制代码

# hpa-order-service.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service-hpa
  namespace: juwatech-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

需在应用中暴露 /actuator/prometheus 端点（Spring Boot Actuator）：

java 复制代码

// application-prod.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: order-service

ResourceQuota 与 LimitRange 控制总量

防止某环境过度占用集群资源：

yaml 复制代码

# quota-prod.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-compute-quota
  namespace: juwatech-prod
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.cpu: "80"
    limits.memory: 160Gi
    pods: "100"

---
apiVersion: v1
kind: LimitRange
metadata:
  name: prod-limits
  namespace: juwatech-prod
spec:
  limits:
  - default:
      cpu: "1"
      memory: "2Gi"
    defaultRequest:
      cpu: "200m"
      memory: "512Mi"
    type: Container

Pod 拓扑分布与高可用调度

确保关键服务跨可用区部署，避免单点故障：

yaml 复制代码

# topology-spread-constraints 示例
spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: commission-calc
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: commission-calc

同时，为批处理任务（如每日返利结算）设置低优先级：

yaml 复制代码

apiVersion: batch/v1
kind: Job
metadata:
  name: daily-commission-job
spec:
  template:
    spec:
      priorityClassName: low-priority  # 需提前创建 PriorityClass
      containers:
      - name: worker
        image: registry.juwatech.cn/commission-worker:1.0

yaml 复制代码

# priority-class.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 1000
globalDefault: false
description: "Low priority for batch jobs"

ConfigMap 与 Secret 环境解耦

敏感配置通过 Secret 注入，非敏感配置用 ConfigMap：

yaml 复制代码

apiVersion: v1
kind: ConfigMap
metadata:
  name: order-service-config
  namespace: juwatech-prod
data:
  application.yml: |
    taobao:
      app-key: ${TAOBAO_APP_KEY}
      secret: ${TAOBAO_SECRET}
    redis:
      host: redis-prod.juwatech.cn

---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: juwatech-prod
type: Opaque
data:
  username: dXNlcg==  # base64 encoded
  password: cGFzc3dvcmQ=

Deployment 中挂载：

yaml 复制代码

envFrom:
- configMapRef:
    name: order-service-config
- secretRef:
    name: db-credentials

本文著作权归聚娃科技省赚客app开发者团队，转载请注明出处！