deepspeed docker集群实现多机多卡训练----问题记录及解决方案资源汇总

. Docker中实现Deepspeed多机多卡训练

• 【掘金-雨田君的记事本】docker容器中deepspeed多机多卡集群分布式训练大模型

. 问题记录及解决方案资源汇总

• 问题1：deepspeed socketStartConnect: Connect to 172.18.0.3<54379> failed : Software caused connection abort

  有效方案：【博客园-高颜值的杀生丸】deepspeed 训练多机多卡报错 ncclSystemError Last error

• 问题2：NCCL WARN Error while creating shared memory segment

  有效方案：【简书-Aiah_Wang】NCCL分布式训练报错

• 问题3：docker swarm: Error response from daemon: rpc error: code = Unavailable desc = connection error

  有效方案：【CSDN-鳄鱼儿】Docker Swarm 解决报错Error response from daemon: rpc error: code = Unavailable desc = connection error:

• 问题4：ImportError: /root/.cache/torch_extensions/py310_cu121/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory

  有效方案：【Github】[BUG][Upstream] py310_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory #2