DeepSpeed 配置文件（DeepSpeed Configuration Files）详解：中英文解释

中文版

本文详细介绍 DeepSpeed 配置文件，结合 4 卡 3090 的实际使用场景，重点解释各个参数的含义，并提供应对爆显存的方案。

---

DeepSpeed 配置文件详解：从基础到实战

DeepSpeed 是用于加速大规模分布式训练的重要工具，其灵活的配置文件是实现高效训练的关键。在本篇博客中，我们将深入解读 DeepSpeed 配置文件的结构和关键参数，结合 4 卡 3090 的实际训练场景，探讨如何优化配置，解决爆显存问题。

---

1. 配置文件的结构

DeepSpeed 的配置文件一般以 JSON 格式定义，包括以下几个核心部分：

• bf16/fp16 配置：决定是否启用混合精度训练。
• ZeRO 优化配置：用于控制内存优化策略。
• 训练相关参数：例如批量大小、梯度累积步数等。

以下是一个典型的配置文件示例：

{
    "bf16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 2,
        "overlap_comm": true,
        "contiguous_gradients": false,
        "reduce_bucket_size": 5e5,
        "sub_group_size": 5e5
    },
    "gradient_accumulation_steps": 4,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_clipping": 1.0
}

---

2. 关键参数解析

bf16.enabled

• 含义：启用 BF16 混合精度训练。
• 影响：显著减少显存占用，提升训练速度。

zero_optimization.stage

• 含义：指定 ZeRO 优化的阶段。
  • Stage 1：优化梯度存储。
  • Stage 2：进一步优化优化器状态存储。
  • Stage 3：支持模型分片。
• 推荐：对于 4 卡 3090，优先选择 Stage 2 ，在显存允许的情况下使用 Stage 3。

overlap_comm

• 含义：启用通信与计算的重叠，减少通信开销。
• 建议：在多卡场景中始终开启。

contiguous_gradients

• 含义：是否在内存中存储连续梯度。
• 优点：开启后可减少内存碎片化，提高通信效率。
• 缺点：增加显存开销。
• 建议：若显存不足，可将其设置为 false。

reduce_bucket_size

• 含义：定义一次通信中参数分片的最大大小。
• 单位：字节。
• 默认值：5e6（即 5 MB）。
• 调整：
  • 若显存不足，减小值至 1e5 或 5e5。
  • 如果通信瓶颈明显，可适当增大值。

sub_group_size

• 含义：设置通信子组的参数分片大小。
• 默认值：1e8（即 100 MB）。
• 调整：
  • 小模型：5e5 或更低。
  • 大模型：可根据显存容量调试，通常 1e6 至 1e7。

gradient_accumulation_steps

• 含义：设置梯度累积步数，减少单步的显存压力。
• 建议：逐步增加值（如从 4 到 8），但需注意总批量大小的变化。

train_micro_batch_size_per_gpu

• 含义：每张 GPU 的微批量大小。
• 建议：在显存不足时减小，如从 4 降为 1。

gradient_clipping

• 含义：限制梯度范数，防止梯度爆炸。
• 推荐值：1.0。

---

3. 针对 4 卡 3090 的优化建议

• 显存不足问题解决方法：

  1. 减小 reduce_bucket_size 和 sub_group_size：

     "reduce_bucket_size": 1e5,
     "sub_group_size": 5e5

  2. 降低 train_micro_batch_size_per_gpu：

     "train_micro_batch_size_per_gpu": 1

  3. 增大 gradient_accumulation_steps：

     "gradient_accumulation_steps": 8

  4. 禁用 contiguous_gradients：

     "contiguous_gradients": false

• 检查 NCCL 环境变量 ：

  确保以下变量已正确设置，避免通信问题导致显存不足。

  export NCCL_BLOCKING_WAIT=1
  export NCCL_ASYNC_ERROR_HANDLING=1
  export NCCL_TIMEOUT=10800

• 启用 CPU Offloading（如果必要） ：

  对于显存严重不足的场景，可将部分优化器状态卸载至 CPU。

  "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
  }

---

4. 实验结果分析与日志监控

在训练过程中，通过以下设置获取详细的资源占用信息：

"wall_clock_breakdown": true

并结合 DeepSpeed 的日志分析显存使用、通信效率等关键指标。

---

通过合理配置 DeepSpeed 配置文件，结合具体的硬件资源和任务需求，可以显著提升训练效率，减少显存压力。

英文版

This article is about explaining DeepSpeed configuration files , focusing on practical usage with a 4x 3090 GPU setup. This includes a breakdown of key parameters like contiguous_gradients, reduce_bucket_size, and sub_group_size, as well as solutions for handling out-of-memory (OOM) errors.

---

DeepSpeed Configuration Files: A Comprehensive Guide

DeepSpeed offers advanced optimization features like ZeRO (Zero Redundancy Optimizer) to enable efficient large-scale model training. This post will delve into configuring DeepSpeed for optimal performance, with examples and tips tailored to a 4x NVIDIA 3090 GPU setup.

---

1. Key Parameters in a DeepSpeed Configuration File

Below is an example configuration file for ZeRO Stage 2 optimization, designed for fine-tuning large models:

{
    "zero_optimization": {
        "stage": 2,
        "overlap_comm": true,
        "contiguous_gradients": false,
        "reduce_bucket_size": 5e5,
        "sub_group_size": 5e5
    },
    "gradient_accumulation_steps": 4,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_clipping": 1.0
}

Let's break down the parameters:

(1) zero_optimization.stage

• Defines the ZeRO optimization stage:
  • Stage 2: Optimizes optimizer states and gradients across GPUs, reducing memory usage.
  • Use Stage 3 for more aggressive memory savings by offloading parameters to CPU, if applicable.

(2) overlap_comm

• Default: true
• Enables overlapping communication with computation, improving efficiency during distributed training.

(3) contiguous_gradients

• Default: false
• When true, all gradients are stored contiguously in memory.
  • Benefit: Faster gradient reductions.
  • Drawback: Increases memory usage.
  • Recommendation: Set to false if facing OOM issues.

(4) reduce_bucket_size

• Defines the size of gradient buckets for all-reduce operations.
  • Smaller values (e.g., 5e5) reduce memory pressure but may slightly slow down training.
  • Larger values improve speed but require more memory.

(5) sub_group_size

• Controls sub-grouping of gradients during communication.
  • Default: A large value (e.g., 1e9), meaning no sub-grouping.
  • Recommendation: Reduce to 5e5 or lower for better memory efficiency.

(6) gradient_accumulation_steps

• Number of steps to accumulate gradients before performing a backward pass.
  • Higher values effectively increase the batch size without increasing per-GPU memory load.

(7) train_micro_batch_size_per_gpu

• Batch size per GPU per step.
  • Recommendation: Start with a small value (e.g., 1) and scale up gradually.

---

2. Handling Out-of-Memory (OOM) Errors

Training large models like Google Gemma-2-2B on GPUs with limited memory (24 GB, such as NVIDIA 3090) often results in OOM errors. Here are optimization strategies:

(1) Reduce train_micro_batch_size_per_gpu

• Start with 1 and only increase if memory allows.

(2) Lower reduce_bucket_size and sub_group_size

• Decrease both to 1e5 or 5e4. This reduces the memory footprint during gradient reduction at the cost of slightly increased communication overhead.

(3) Enable offload_optimizer or offload_param (for ZeRO Stage 3)

• Offload optimizer states or parameters to CPU if memory remains insufficient.

• Example configuration for optimizer offloading:

  {
      "zero_optimization": {
          "stage": 3,
          "offload_optimizer": {
              "device": "cpu",
              "pin_memory": true
          }
      }
  }

(4) Use Gradient Checkpointing

• Activates checkpointing for intermediate activations to save memory during backpropagation.

  from deepspeed.runtime.activation_checkpointing import checkpointing_config
  checkpointing_config(
      partition_activations=True,
      contiguous_memory_optimization=False
  )

(5) Mixed Precision Training (bf16 or fp16)

• Use bf16 for better memory efficiency with minimal precision loss.

(6) Increase gradient_accumulation_steps

• Accumulate gradients over more steps to reduce the batch size processed per GPU.

(7) Reduce max_seq_length

• Shorten sequence length (e.g., 512 or 768 tokens) to decrease memory usage.

---

3. Practical Example: Fine-Tuning on 4x NVIDIA 3090 GPUs

The following accelerate command illustrates how to combine the above settings for fine-tuning a large model:

accelerate launch \
    --mixed_precision bf16 \
    --num_machines 1 \
    --num_processes 4 \
    --machine_rank 0 \
    --main_process_ip 127.0.0.1 \
    --main_process_port 29400 \
    --use_deepspeed \
    --deepspeed_config_file configs/ds_config.json \
    --model_name_or_path google/gemma-2-2b \
    --tokenizer_name google/gemma-2-2b \
    --max_seq_length 768 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 4 \
    --learning_rate 5e-6 \
    --num_train_epochs 1 \
    --output_dir output/sft_gemma2

---

4. Debugging Tips

• Enable Detailed Logs: Set wall_clock_breakdown: true in the config file to identify bottlenecks.

• NCCL Tuning: Add environment variables to handle communication errors:

  export NCCL_BLOCKING_WAIT=1
  export NCCL_ASYNC_ERROR_HANDLING=1

---

Conclusion

DeepSpeed's configuration is highly flexible, but tuning requires balancing memory efficiency and computational speed. By adjusting parameters like reduce_bucket_size, gradient_accumulation_steps, and leveraging ZeRO's offloading capabilities, you can effectively train large models even on memory-constrained GPUs like the NVIDIA 3090.

后记

2024年11月27日22点08分于上海，基于GPT4o大模型。