deepspeed docker集群实现多机多卡训练----问题记录及解决方案资源汇总

XMoyas2024-06-28 15:53

问题1：deepspeed socketStartConnect: Connect to 172.18.0.3<54379> failed : Software caused connection abort

有效方案：【博客园-高颜值的杀生丸】deepspeed 训练多机多卡报错 ncclSystemError Last error
问题2：NCCL WARN Error while creating shared memory segment

有效方案：【简书-Aiah_Wang】NCCL分布式训练报错
问题3：docker swarm: Error response from daemon: rpc error: code = Unavailable desc = connection error

有效方案：【CSDN-鳄鱼儿】Docker Swarm 解决报错Error response from daemon: rpc error: code = Unavailable desc = connection error:
问题4：ImportError: /root/.cache/torch_extensions/py310_cu121/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory

有效方案：【Github】 $BUG$ $Upstream$ py310_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory #2