llama.cpp部署(windows)

一、下载源码和模型

下载源码和模型
复制代码
# 下载源码
git clone https://github.com/ggerganov/llama.cpp.git

# 下载llama-7b模型
git clone https://www.modelscope.cn/skyline2006/llama-7b.git
查看cmake版本:
复制代码
D:\pyworkspace\llama_cpp\llama.cpp\build>cmake --version
cmake version 3.22.0-rc2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

二、开始build

复制代码
# 进入llama.cpp目录
mkdir build
cd build
cmake ..

build信息

复制代码
D:\pyworkspace\llama_cpp\llama.cpp\build>cmake ..
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.22631.
-- The C compiler identification is MSVC 19.29.30137.0
-- The CXX compiler identification is MSVC 19.29.30137.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: D:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: D:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: D:/Git/Git/cmd/git.exe (found version "2.29.2.windows.2")
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: AMD64
-- CMAKE_GENERATOR_PLATFORM:
-- x86 detected
-- Performing Test HAS_AVX_1
-- Performing Test HAS_AVX_1 - Success
-- Performing Test HAS_AVX2_1
-- Performing Test HAS_AVX2_1 - Success
-- Performing Test HAS_FMA_1
-- Performing Test HAS_FMA_1 - Success
-- Performing Test HAS_AVX512_1
-- Performing Test HAS_AVX512_1 - Failed
-- Performing Test HAS_AVX512_2
-- Performing Test HAS_AVX512_2 - Failed
-- Configuring done
-- Generating done
-- Build files have been written to: D:/pyworkspace/llama_cpp/llama.cpp/build

本地使用Realease会出现报错,修改为Debug进行build,这里会使用到visual studio进行build

复制代码
cmake --build . --config Debug

build信息

复制代码
D:\pyworkspace\llama_cpp\llama.cpp\build>cmake --build . --config Debug
用于 .NET Framework 的 Microsoft (R) 生成引擎版本 16.11.2+f32259642
版权所有(C) Microsoft Corporation。保留所有权利。

  Checking Build System
  Generating build details from Git
  -- Found Git: D:/Git/Git/cmd/git.exe (found version "2.29.2.windows.2")
  Building Custom Rule D:/pyworkspace/llama_cpp/llama.cpp/common/CMakeLists.txt
  build-info.cpp
  build_info.vcxproj -> D:\pyworkspace\llama_cpp\llama.cpp\build\common\build_info.dir\Debug\build_info.lib
  Building Custom Rule D:/pyworkspace/llama_cpp/llama.cpp/CMakeLists.txt
  ggml.c

在我本地D:\pyworkspace\llama_cpp\llama.cpp\build\bin\Debug目录下面产生了quantize.exe和main.exe等

三、量化和推理

安装相关python依赖

复制代码
python -m pip install -r requirements.txt

将下载好的llama-7b模型放入models目录下,并执行命令,会在llama-7b目录下面产生ggml-model-f16.gguf文件

复制代码
python convert.py models/llama-7b/

对产生的文件进行量化

复制代码
D:\pyworkspace\llama_cpp\llama.cpp\build\bin\Debug\quantize.exe ./models/llama-7b/ggml-model-f16.gguf ./models/llama-7b/ggml-model-q4_0.gguf q4_0

进行推理

复制代码
D:\pyworkspace\llama_cpp\llama.cpp\build\bin\Debug\main.exe -m ./models/llama-7b/ggml-model-q4_0.gguf -n 256 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt
相关推荐
临街的小孩2 小时前
Docker 容器访问宿主机 Ollama 服务配置教程
llama·argflow
AIGC_北苏3 小时前
EvalScope模型压力测试实战
人工智能·语言模型·模型评估·框架评估
IT小哥哥呀3 小时前
论文见解:REACT:在语言模型中协同推理和行动
前端·人工智能·react.js·语言模型
鸿蒙小白龙3 小时前
OpenHarmony平台大语言模型本地推理:llama深度适配与部署技术详解
人工智能·语言模型·harmonyos·鸿蒙·鸿蒙系统·llama·open harmony
jerryinwuhan3 小时前
对图片进行解释的大语言模型
人工智能·语言模型·自然语言处理
MichaelIp4 小时前
基于MCP协议的多AGENT文章自动编写系统
语言模型·langchain·prompt·ai写作·llamaindex·langgraph·mcp
FserSuN4 小时前
构建基于大语言模型的智能数据可视化分析工具的学习总结
学习·信息可视化·语言模型
MasonYyp19 小时前
拖拽式构建智能体的框架
语言模型
强哥之神1 天前
从零理解 KV Cache:大语言模型推理加速的核心机制
人工智能·深度学习·机器学习·语言模型·llm·kvcache
悟乙己1 天前
大型语言模型(LLM)文本中提取结构化信息:LangExtract(一)
人工智能·语言模型·自然语言处理