大模型训练中CUDA out of memory的几种解决方法

常用:

  1. 减小batch size

  2. 如果需要较大的batch size来达到预期训练指标效果,则使用gradient accumulation来累积梯度,达到等价效果的同时,占用显存小;

  3. 使用低精度做训练。(mixed precision training). FP16+FP32

  4. 减小训练数据样本的长度

  5. Model并行或者Pipeline并行,把模型拆分至多卡

不常用:

  1. 释放闲置内存;torch.cuda.empty_cache()

  2. 缓解PyTorch的显存碎片;PYTORCH_CUDA_ALLOC_CONF和max_split_size_mb

7.使用更小的模型

  1. 增大GPU显存

  2. 使用nvidia-smi看到该GPU上的其他进程;可杀则杀;

Perplexity 查询结果:

Here are several solutions to resolve the "CUDA out of memory" error when training large language models (LLMs) using SFT (Supervised Fine-Tuning):

  1. Reduce batch size: Lowering the batch size during training can significantly reduce the memory requirements. For example, you could set the batch size to 1 and use gradient accumulation to accumulate gradients over multiple batches before updating the model weights.
  2. Use gradient accumulation: Gradient accumulation allows you to accumulate gradients over multiple batches before updating the model weights. This effectively increases the batch size without increasing the memory usage. Be sure to divide the loss by the number of gradient accumulation steps to maintain the correct gradient scale.
  3. Enable mixed precision training: Training the model in mixed precision (FP16) can reduce the memory usage compared to full precision (FP32). PyTorch provides the torch.cuda.amp package for automatic mixed precision training.
  4. Reduce input sequence length: If your input texts are long, consider truncating or splitting them into shorter sequences to reduce the memory usage during tokenization and model inference.
  5. Free GPU memory: Call torch.cuda.empty_cache() to free up any unused GPU memory allocated by PyTorch. This might not help much, but it's worth a try.
  6. Set PYTORCH_CUDA_ALLOC_CONF environment variable: You can try setting the PYTORCH_CUDA_ALLOC_CONF environment variable with max_split_size_mb to avoid memory fragmentation. Experiment with different values based on your GPU memory.
  7. Use a smaller model: If possible, consider using a smaller version of the LLM that has fewer parameters. Smaller models generally require less memory during training.
  8. Increase GPU memory: If you have access to a GPU with more memory, consider using it to accommodate the memory requirements of the LLM training.
  9. Distribute training across multiple GPUs: Splitting the training across multiple GPUs can reduce the memory usage per GPU. However, this approach requires additional setup and coordination.
  10. Monitor GPU memory usage: Use nvidia-smi to monitor the GPU memory usage and identify any other processes consuming GPU memory. Terminate unnecessary processes to free up memory
相关推荐
kadog22 分钟前
PubMed PDF下载 cloudpmc-viewer-pow逆向
前端·javascript·人工智能·爬虫·pdf
亿坊电商41 分钟前
AI数字人多模态技术如何提升用户体验?
人工智能·ux·ai数字人
不吃香菜?1 小时前
PyTorch 实现食物图像分类实战:从数据处理到模型训练
人工智能·深度学习
Jackilina_Stone1 小时前
【论文阅读】平滑量化:对大型语言模型进行准确高效的训练后量化
人工智能·llm·量化·论文阅读笔记
-曾牛2 小时前
企业级AI开发利器:Spring AI框架深度解析与实战
java·人工智能·python·spring·ai·rag·大模型应用
TMT星球2 小时前
商汤绝影生成式AI汽车新品亮相上海车展,引领AI汽车创新潮流
人工智能·汽车
Light602 小时前
智启未来:深度解析Python Transformers库及其应用场景
开发语言·python·深度学习·自然语言处理·预训练模型·transformers库 |·|应用场景
爱的叹息2 小时前
DeepSeek 大模型 + LlamaIndex + MySQL 数据库 + 知识文档 实现简单 RAG 系统
数据库·人工智能·mysql·langchain
数据智能老司机2 小时前
构建具备自主性的人工智能系统——在生成式人工智能系统中构建信任
深度学习·llm·aigc
PeterOne2 小时前
Trae MCP + Obsidian 集成如何缓解开发者的时间损耗
人工智能·trae