text-generation-webui加载codellama报错DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。

使用text-generation-webui加载codellama,报错:

python 复制代码
Traceback (most recent call last):

File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1353, in _get_module


return importlib.import_module("." + module_name, self.__name__)
File "D:\Anaconda\Anaconda\envs\codellama\lib\importlib_init_.py", line 126, in import_module


return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import

File "", line 1027, in _find_and_load

File "", line 1006, in _find_and_load_unlocked

File "", line 688, in _load_unlocked

File "", line 883, in exec_module

File "", line 241, in _call_with_frames_removed

File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\llama\modeling_llama.py", line 48, in


from flash_attn import flash_attn_func, flash_attn_varlen_func
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn_init_.py", line 3, in


from flash_attn.flash_attn_interface import (
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\flash_attn\flash_attn_interface.py", line 8, in


import flash_attn_2_cuda as flash_attn_cuda
ImportError: DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "E:\模型\text-generation-webui\text-generation-webui\modules\ui_model_menu.py", line 209, in load_model_wrapper


shared.model, shared.tokenizer = load_model(shared.model_name, loader)
File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 85, in load_model


output = load_func_map[loader](model_name)
File "E:\模型\text-generation-webui\text-generation-webui\modules\models.py", line 155, in huggingface_loader


model = LoaderClass.from_pretrained(path_to_model, **params)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 565, in from_pretrained


model_class = _get_model_class(config, cls._model_mapping)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 387, in _get_model_class


supported_models = model_mapping[type(config)]
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 740, in getitem


return self._load_attr_from_module(model_type, model_name)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 754, in _load_attr_from_module


return getattribute_from_module(self._modules[module_name], attr)
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 698, in getattribute_from_module


if hasattr(module, attr):
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1343, in getattr


module = self._get_module(self._class_to_module[name])
File "C:\Users\Ma\AppData\Roaming\Python\Python310\site-packages\transformers\utils\import_utils.py", line 1355, in _get_module


raise RuntimeError(
RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):

DLL load failed while importing flash_attn_2_cuda: 找不到指定的模块。

一开始排查是以为transformers的版本不对,先确定了transformers的版本,transformers的版本应该大于4.35.0

把transformers升级为4.35.0后仍然报错

接着排查cuda和torch的版本

最后发现是cuda版本与torch版本不匹配

python 复制代码
>>> print(torch.version.cuda) # 检查CUDA版本
>>> 11.8

控制台运行nvcc --version :

输出:

python 复制代码
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

最后解决:

先卸载原本的torch:

python 复制代码
pip uninstall torch torchvision torchaudio

然后安装12.1的:

python 复制代码
pip install torch torchvision torchaudio -f https://download.pytorch.org/whl/cu121/torch_stable.html

最后加载成功codellama

相关推荐
Csvn12 小时前
如果明天供应商涨价 3 倍?三层解耦,换模型只改一行配置(C05)
人工智能·aigc·agent
在所不辞兄13 小时前
为什么数字孪生驱动的产线标定需要在线更新机制,而不能一次标定后长期沿用?
人工智能·神经网络·其他·算法·机器学习
锋行天下13 小时前
langgraph 全栈项目(三)
前端·人工智能·后端
黄啊码14 小时前
【黄啊码】最近面了十几个想转 FDE 的程序员,一个都没招
人工智能
锋行天下14 小时前
langgraph 全栈项目(二)
前端·人工智能·后端
大模型真好玩14 小时前
用 Seedance 2.5 Flash 实现1分钟AI短剧,满足孩子的超级英雄对战梦
人工智能·aigc·deepseek
FengY_HYY15 小时前
从人类组织到人类&大模型组织,我的位置在哪里?
人工智能·程序员·llm
Flynn Liang15 小时前
大模型成本优化:基于 Redis 的可验证语义缓存架构
人工智能·redis·缓存·架构·ai agent·智能体
锋行天下15 小时前
langgraph全栈项目(一)
前端·人工智能·后端
程序员cxuan15 小时前
不得了,Google 造反了,Opus 5.5 又被干了
人工智能·后端·程序员