【CUDA 编译 bug】ld: cannot find -lcudart

PLUS_WAVE2025-04-26 8:13

我们使用 Conda 安装 pytorch 和 CUDA 环境之后，要用 Conda 的CUDA环境进行某个库编译时，出现了bug：

bash 复制代码 
/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/compiler_compat/ld: cannot find -lcudart: No such file or directory
      collect2: error: ld returned 1 exit status
      error: command '/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/bin/g++' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for diff_gaussian_rasterization_32d
  Running setup.py clean for diff_gaussian_rasterization_32d
Failed to build diff_gaussian_rasterization_32d
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (diff_gaussian_rasterization_32d)
❯ which nvcc
/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/bin/nvcc
❯ echo $CUDA_HOME
/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar
❯ echo $PATH
/home/xxxx/local/bin:/home/xxxx/local/bin:/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/bin:/mnt/data/home/xxxx/miniforge3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
❯ echo $LD_LIBRARY_PATH
/mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/lib:

去探查发现，这里的软链接出了问题：

bash 复制代码 
❯ ls /mnt/data/home/xxxx/miniforge3/envs/GAGAvatar/lib

libcudart.so  -> libcudart.so.12.1.55
libcudart.so.12
libcudart.so.12.1.105

继续探究发现，安装Pytorch时会安装 cuda-cudart=12.1.105

以下是按照Pytorch时会安装的所有以 pytorch 、 nvidia 为 channel 的包：

bash 复制代码 
  + pytorch-mutex         1.0  cuda                          pytorch     Cached
  + libcublas       12.1.0.26  0                             nvidia      Cached
  + libcufft         11.0.2.4  0                             nvidia      Cached
  + libcusolver     11.4.4.55  0                             nvidia      Cached
  + libcusparse     12.0.2.55  0                             nvidia      Cached
  + libnpp          12.0.2.50  0                             nvidia      Cached
  + cuda-cudart      12.1.105  0                             nvidia      Cached
  + cuda-nvrtc       12.1.105  0                             nvidia      Cached
  + libnvjitlink     12.1.105  0                             nvidia      Cached
  + libnvjpeg       12.1.1.14  0                             nvidia      Cached
  + cuda-cupti       12.1.105  0                             nvidia      Cached
  + cuda-nvtx        12.1.105  0                             nvidia      Cached
  + ffmpeg                4.3  hf484d3e_0                    pytorch     Cached
  + libjpeg-turbo       2.0.0  h9bf148f_0                    pytorch     Cached
  + cuda-version         12.6  3                             nvidia      Cached
  + libcurand       10.3.7.77  0                             nvidia      Cached
  + libcufile        1.11.1.6  0                             nvidia      Cached
  + cuda-opencl       12.6.77  0                             nvidia      Cached
  + cuda-libraries     12.1.0  0                             nvidia      Cached
  + cuda-runtime       12.1.0  0                             nvidia      Cached
  + pytorch-cuda         12.1  ha16c6d3_6                    pytorch     Cached
  + pytorch             2.4.1  py3.12_cuda12.1_cudnn9.1.0_0  pytorch     Cached
  + torchtriton         3.0.0  py312                         pytorch     Cached
  + torchaudio          2.4.1  py312_cu121                   pytorch     Cached
  + torchvision        0.19.1  py312_cu121                   pytorch     Cached

而这是安装 cuda-toolkit-12.1.0 的包：

bash 复制代码 
  + cuda-documentation           12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvml-dev                12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + libnvvm-samples              12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cccl                    12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-driver-dev              12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-profiler-api            12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cudart                  12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvrtc                   12.1.55  0      nvidia/label/cuda-12.1.0       21MB
  + cuda-opencl                  12.1.56  0      nvidia/label/cuda-12.1.0       11kB
  + libcublas                  12.1.0.26  0      nvidia/label/cuda-12.1.0     Cached
  + libcufft                    11.0.2.4  0      nvidia/label/cuda-12.1.0     Cached
  + libcufile                   1.6.0.25  0      nvidia/label/cuda-12.1.0      782kB
  + libcurand                  10.3.2.56  0      nvidia/label/cuda-12.1.0       54MB
  + libcusolver                11.4.4.55  0      nvidia/label/cuda-12.1.0     Cached
  + libcusparse                12.0.2.55  0      nvidia/label/cuda-12.1.0     Cached
  + libnpp                     12.0.2.50  0      nvidia/label/cuda-12.1.0     Cached
  + libnvjitlink                 12.1.55  0      nvidia/label/cuda-12.1.0       18MB
  + libnvjpeg                  12.1.0.39  0      nvidia/label/cuda-12.1.0        3MB
  + cuda-cupti                   12.1.62  0      nvidia/label/cuda-12.1.0        5MB
  + cuda-cuobjdump               12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cuxxfilt                12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvcc                    12.1.66  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvprune                 12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-gdb                     12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvdisasm                12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvprof                  12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvtx                    12.1.66  0      nvidia/label/cuda-12.1.0       58kB
  + cuda-sanitizer-api           12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nsight                  12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + nsight-compute           2023.1.0.15  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cudart-dev              12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvrtc-dev               12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-opencl-dev              12.1.56  0      nvidia/label/cuda-12.1.0     Cached
  + libcublas-dev              12.1.0.26  0      nvidia/label/cuda-12.1.0     Cached
  + libcufft-dev                11.0.2.4  0      nvidia/label/cuda-12.1.0     Cached
  + gds-tools                   1.6.0.25  0      nvidia/label/cuda-12.1.0     Cached
  + libcufile-dev               1.6.0.25  0      nvidia/label/cuda-12.1.0     Cached
  + libcurand-dev              10.3.2.56  0      nvidia/label/cuda-12.1.0     Cached
  + libcusolver-dev            11.4.4.55  0      nvidia/label/cuda-12.1.0     Cached
  + libcusparse-dev            12.0.2.55  0      nvidia/label/cuda-12.1.0     Cached
  + libnpp-dev                 12.0.2.50  0      nvidia/label/cuda-12.1.0     Cached
  + libnvjitlink-dev             12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + libnvjpeg-dev              12.1.0.39  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-libraries                12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cupti-static            12.1.62  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-compiler                 12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvvp                    12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-command-line-tools       12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nsight-compute           12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-cudart-static           12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-nvrtc-static            12.1.55  0      nvidia/label/cuda-12.1.0     Cached
  + libcublas-static           12.1.0.26  0      nvidia/label/cuda-12.1.0     Cached
  + libcufft-static             11.0.2.4  0      nvidia/label/cuda-12.1.0     Cached
  + libcufile-static            1.6.0.25  0      nvidia/label/cuda-12.1.0     Cached
  + libcurand-static           10.3.2.56  0      nvidia/label/cuda-12.1.0     Cached
  + libcusolver-static         11.4.4.55  0      nvidia/label/cuda-12.1.0     Cached
  + libcusparse-static         12.0.2.55  0      nvidia/label/cuda-12.1.0     Cached
  + libnpp-static              12.0.2.50  0      nvidia/label/cuda-12.1.0     Cached
  + libnvjpeg-static           12.1.0.39  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-libraries-dev            12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-libraries-static         12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-visual-tools             12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-tools                    12.1.0  0      nvidia/label/cuda-12.1.0     Cached
  + cuda-toolkit                  12.1.0  0      nvidia/label/cuda-12.1.0     Cached

这是安装 cuda-toolkit-12.1.1 的包：

bash 复制代码 
  + cuda-documentation         12.1.105  0      nvidia/label/cuda-12.1.1       91kB
  + cuda-nvml-dev              12.1.105  0      nvidia/label/cuda-12.1.1       87kB
  + libnvvm-samples            12.1.105  0      nvidia/label/cuda-12.1.1       33kB
  + cuda-cccl                  12.1.109  0      nvidia/label/cuda-12.1.1        1MB
  + cuda-driver-dev            12.1.105  0      nvidia/label/cuda-12.1.1       17kB
  + cuda-profiler-api          12.1.105  0      nvidia/label/cuda-12.1.1       19kB
  + cuda-cudart                12.1.105  0      nvidia/label/cuda-12.1.1     Cached
  + cuda-nvrtc                 12.1.105  0      nvidia/label/cuda-12.1.1     Cached
  + cuda-opencl                12.1.105  0      nvidia/label/cuda-12.1.1       11kB
  + libcublas                  12.1.3.1  0      nvidia/label/cuda-12.1.1      367MB
  + libcufft                  11.0.2.54  0      nvidia/label/cuda-12.1.1      108MB
  + libcufile                   1.6.1.9  0      nvidia/label/cuda-12.1.1      783kB
  + libcurand                10.3.2.106  0      nvidia/label/cuda-12.1.1       54MB
  + libcusolver              11.4.5.107  0      nvidia/label/cuda-12.1.1      116MB
  + libcusparse              12.1.0.106  0      nvidia/label/cuda-12.1.1      177MB
  + libnpp                    12.1.0.40  0      nvidia/label/cuda-12.1.1      147MB
  + libnvjitlink               12.1.105  0      nvidia/label/cuda-12.1.1     Cached
  + libnvjpeg                  12.2.0.2  0      nvidia/label/cuda-12.1.1        3MB
  + cuda-cupti                 12.1.105  0      nvidia/label/cuda-12.1.1     Cached
  + cuda-cuobjdump             12.1.111  0      nvidia/label/cuda-12.1.1      245kB
  + cuda-cuxxfilt              12.1.105  0      nvidia/label/cuda-12.1.1      302kB
  + cuda-nvcc                  12.1.105  0      nvidia/label/cuda-12.1.1       55MB
  + cuda-nvprune               12.1.105  0      nvidia/label/cuda-12.1.1       67kB
  + cuda-gdb                   12.1.105  0      nvidia/label/cuda-12.1.1        6MB
  + cuda-nvdisasm              12.1.105  0      nvidia/label/cuda-12.1.1       50MB
  + cuda-nvprof                12.1.105  0      nvidia/label/cuda-12.1.1        5MB
  + cuda-nvtx                  12.1.105  0      nvidia/label/cuda-12.1.1     Cached
  + cuda-sanitizer-api         12.1.105  0      nvidia/label/cuda-12.1.1       18MB
  + cuda-nsight                12.1.105  0      nvidia/label/cuda-12.1.1      119MB
  + nsight-compute           2023.1.1.4  0      nvidia/label/cuda-12.1.1      808MB
  + cuda-cudart-dev            12.1.105  0      nvidia/label/cuda-12.1.1      381kB
  + cuda-nvrtc-dev             12.1.105  0      nvidia/label/cuda-12.1.1       12kB
  + cuda-opencl-dev            12.1.105  0      nvidia/label/cuda-12.1.1       59kB
  + libcublas-dev              12.1.3.1  0      nvidia/label/cuda-12.1.1       76kB
  + libcufft-dev              11.0.2.54  0      nvidia/label/cuda-12.1.1       14kB
  + gds-tools                   1.6.1.9  0      nvidia/label/cuda-12.1.1       43MB
  + libcufile-dev               1.6.1.9  0      nvidia/label/cuda-12.1.1       13kB
  + libcurand-dev            10.3.2.106  0      nvidia/label/cuda-12.1.1      460kB
  + libcusolver-dev          11.4.5.107  0      nvidia/label/cuda-12.1.1       51kB
  + libcusparse-dev          12.1.0.106  0      nvidia/label/cuda-12.1.1      178MB
  + libnpp-dev                12.1.0.40  0      nvidia/label/cuda-12.1.1      525kB
  + libnvjitlink-dev           12.1.105  0      nvidia/label/cuda-12.1.1       15MB
  + libnvjpeg-dev              12.2.0.2  0      nvidia/label/cuda-12.1.1       13kB
  + cuda-libraries               12.1.1  0      nvidia/label/cuda-12.1.1        2kB
  + cuda-cupti-static          12.1.105  0      nvidia/label/cuda-12.1.1       12MB
  + cuda-compiler                12.1.1  0      nvidia/label/cuda-12.1.1        1kB
  + cuda-nvvp                  12.1.105  0      nvidia/label/cuda-12.1.1      120MB
  + cuda-command-line-tools      12.1.1  0      nvidia/label/cuda-12.1.1        1kB
  + cuda-nsight-compute          12.1.1  0      nvidia/label/cuda-12.1.1        1kB
  + cuda-cudart-static         12.1.105  0      nvidia/label/cuda-12.1.1      948kB
  + cuda-nvrtc-static          12.1.105  0      nvidia/label/cuda-12.1.1       18MB
  + libcublas-static           12.1.3.1  0      nvidia/label/cuda-12.1.1      389MB
  + libcufft-static           11.0.2.54  0      nvidia/label/cuda-12.1.1      199MB
  + libcufile-static            1.6.1.9  0      nvidia/label/cuda-12.1.1        3MB
  + libcurand-static         10.3.2.106  0      nvidia/label/cuda-12.1.1       55MB
  + libcusolver-static       11.4.5.107  0      nvidia/label/cuda-12.1.1       76MB
  + libcusparse-static       12.1.0.106  0      nvidia/label/cuda-12.1.1      185MB
  + libnpp-static             12.1.0.40  0      nvidia/label/cuda-12.1.1      143MB
  + libnvjpeg-static           12.2.0.2  0      nvidia/label/cuda-12.1.1        3MB
  + cuda-libraries-dev           12.1.1  0      nvidia/label/cuda-12.1.1        2kB
  + cuda-libraries-static        12.1.1  0      nvidia/label/cuda-12.1.1        2kB
  + cuda-visual-tools            12.1.1  0      nvidia/label/cuda-12.1.1        1kB
  + cuda-tools                   12.1.1  0      nvidia/label/cuda-12.1.1        1kB
  + cuda-toolkit                 12.1.1  0      nvidia/label/cuda-12.1.1        2kB

对比发现是 cuda-12.1.1 才对的上CUDA版本12.1的Pytorch。但是我们在安装的时候，先安装CUDA版本12.1的Pytorch，再安装 cuda-12.1.1 会出现冲突问题：

bash 复制代码 
└─ pytorch-cuda is not installable because it requires
   └─ libcublas >=12.1.0.26,<12.1.3.1 , which conflicts with any installable versions previously reported.

也就是说，该死的CUDA版本12.1的Pytorch的 libcublas 需要适配 cuda-toolkit-12.1.0 ，但是其的 cuda-cudart 等库又需要适配 cuda-toolkit-12.1.1

可以看到 pytorch-cuda 强要求 libcublas >=12.1.0.26,<12.1.3.1，我们只好迁就 pytorch，安装12.1.0的CUDA，但是呢！我们可以修改Pytorch官方给出的 nvidia channel 为 nvidia/label/cuda-12.1.0

使用以下命令：

bash 复制代码 
mamba install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia/label/cuda-12.1.0

其就会安装与我们安装的 cuda-toolkit-12.1.0 一样的一些 cuda 库了！

bash 复制代码 
  + pytorch-mutex         1.0  cuda                          pytorch                      Cached
  + libcublas       12.1.0.26  0                             nvidia/label/cuda-12.1.0     Cached
  + libcufft         11.0.2.4  0                             nvidia/label/cuda-12.1.0     Cached
  + libcusolver     11.4.4.55  0                             nvidia/label/cuda-12.1.0     Cached
  + libcusparse     12.0.2.55  0                             nvidia/label/cuda-12.1.0     Cached
  + libnpp          12.0.2.50  0                             nvidia/label/cuda-12.1.0     Cached
  + libnvjpeg       12.1.0.39  0                             nvidia/label/cuda-12.1.0        3MB
  + cuda-cudart       12.1.55  0                             nvidia/label/cuda-12.1.0     Cached
  + cuda-nvrtc        12.1.55  0                             nvidia/label/cuda-12.1.0       21MB
  + cuda-opencl       12.1.56  0                             nvidia/label/cuda-12.1.0       11kB
  + libcufile        1.6.0.25  0                             nvidia/label/cuda-12.1.0      782kB
  + libcurand       10.3.2.56  0                             nvidia/label/cuda-12.1.0       54MB
  + cuda-cupti        12.1.62  0                             nvidia/label/cuda-12.1.0        5MB
  + cuda-nvtx         12.1.66  0                             nvidia/label/cuda-12.1.0       58kB
  + cuda-version         12.1  h1d6eff3_3                    conda-forge                    21kB
  + ffmpeg                4.3  hf484d3e_0                    pytorch                      Cached
  + libjpeg-turbo       2.0.0  h9bf148f_0                    pytorch                      Cached
  + libnvjitlink     12.1.105  hd3aeb46_0                    conda-forge                    16MB
  + cuda-libraries     12.1.0  0                             nvidia/label/cuda-12.1.0     Cached
  + cuda-runtime       12.1.0  0                             nvidia/label/cuda-12.1.0     Cached
  + pytorch-cuda         12.1  ha16c6d3_6                    pytorch                      Cached
  + pytorch             2.4.1  py3.12_cuda12.1_cudnn9.1.0_0  pytorch                      Cached
  + torchtriton         3.0.0  py312                         pytorch                      Cached
  + torchvision        0.19.1  py312_cu121                   pytorch                      Cached
  + torchaudio          2.4.1  py312_cu121                   pytorch                      Cached

到这里，问题就解决了：我们之后要安装 pytorch-cuda 和 cuda-toolkit 时，只需要执行以下命令（顺序应该不重要了）：

bash 复制代码 
mamba install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia/label/cuda-12.1.0
mamba install nvidia/label/cuda-12.1.0::cuda-toolkit -c nvidia/label/cuda-12.1.0

安装 cuda-toolkit 就相当于在安装完 pytorch-cuda 的需要的部分 cuda 库后，进行了补充安装，都是同一个 channel 的当然就不会有问题了

相关推荐
YuforiaCode2 分钟前
第十二届蓝桥杯 2021 C/C++组 直线
c语言·c++·蓝桥杯
子非衣12 分钟前
Windows云主机远程连接提示“出现了内部错误”
服务器·windows
草莓啵啵～1 小时前
搜索二叉树-key的搜索模型
数据结构·c++
共享家95271 小时前
深入理解C++ 中的list容器
c++
孞㐑￥1 小时前
C++11介绍
开发语言·c++·经验分享·笔记
云小逸1 小时前
【QQMusic项目界面开发复习笔记】第二章
c++·qt
lLinkl1 小时前
项目笔记2：post请求是什么，还有什么请求
服务器·网络协议·http
李匠20241 小时前
C++ RPC以及cmake
网络·c++·网络协议·rpc
珹洺2 小时前
Linux操作系统从入门到实战（三）Linux基础指令（上）
linux·运维·服务器
再睡一夏就好2 小时前
Linux常见工具如yum、vim、gcc、gdb的基本使用，以及编译过程和动静态链接的区别
linux·服务器·c语言·c++·笔记
热门推荐
01KGG转MP3工具|非KGM文件|解密音频02从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑03我决定放弃搞 Java 了04Coze扣子平台完整体验和实践（附国内和国际版对比）05YOLOv8入门 | 重要性能衡量指标、训练结果评价及分析及影响mAP的因素【发论文关注的指标】06DeepSeek各版本说明与优缺点分析07yolov8，yolo11，yolo12 服务器训练到部署全流程 笔记08苍穹外卖面试总结09如何在WPS和Word/Excel中直接使用DeepSeek功能10【2025年最新】OpenWrt 更换国内源的指南（图形界面版)