deploy local llm ragflow

CPU >= 4 cores

RAM >= 16 GB

Disk >= 50 GB

Docker >= 24.0.0 & Docker Compose >= v2.26.1

下载docker:

官方下载方式:https://docs.docker.com/desktop/install/ubuntu/

其中 DEB package需要手动下载并传输到服务器

国内下载方式:

https://blog.csdn.net/u011278722/article/details/137673353

Ensure vm.max_map_count >= 262144:

check:

$ sysctl vm.max_map_count

Reset vm.max_map_count to a value at least 262144 if it is not:

$ sudo sysctl -w vm.max_map_count=262144

This change will be reset after a system reboot. To ensure your change remains permanent, add or 
        update the vm.max_map_count value in /etc/sysctl.conf accordingly:
$ vm.max_map_count=262144

Clone the repo:

$ git clone https://github.com/infiniflow/ragflow.git

该步骤需要手动下载并传输,国内无法下载

Build the pre-built Docker images and start up the server:

$ cd ragflow/docker

$ chmod +x ./entrypoint.sh

$ docker compose up -d

这一步也需要手动传输或直接用用源代码build(见最后)

Check the server status after having the server up and running:

$ docker logs -f ragflow-server

The following output confirms a successful launch of the system:


/ __ \ ____ _ ____ _ / // / _ __

/ // // __ // __ // / / // __ | | /| / /

/ , // // // / / // / / // // /| |/ |/ /
/
/ || _ ,/ _ , /// // _
/ | /| _/

/____/

In your web browser, enter the IP address of your server and log in to RAGFlow.

With the default settings, you only need to enter http://IP_OF_YOUR_MACHINE (sans port number) as the default HTTP serving port 80 can be omitted when using the default configurations.

In service_conf.yaml, select the desired LLM factory in user_default_llm and update the API_KEY field with the corresponding API key.

See llm_api_key_setup for more information.

Rebuild:

To build the Docker images from source:

$ git clone https://github.com/infiniflow/ragflow.git

$ cd ragflow/

$ docker build -t infiniflow/ragflow:dev .

$ cd ragflow/docker

$ chmod +x ./entrypoint.sh

$ docker compose up -d

卸载原有cuda和驱动

https://blog.alumik.cn/posts/90/#:\~:text=Use the following command to uninstall a Toolkit,remove --purge '^nvidia-.*' sudo apt-get remove --purge '^libnvidia-.*'

CUDA 和 Nvdia driver安装:

https://blog.hellowood.dev/posts/ubuntu-22-安装-nvdia-显卡驱动和-cuda/

下载Vllm

https://qwen.readthedocs.io/zh-cn/latest/deployment/vllm.html

国内下载model: /Qwen2-7B-Instruct方法:

pip install modelscope

from modelscope import snapshot_download

model_dir = snapshot_download('qwen/Qwen2-7B-Instruct', cache_dir='/home/llmlocal/qwen/qwen/')

运行llm服务器

python -m vllm.entrypoints.openai.api_server --model /home/llmlocal/qwen/qwen/Qwen2-7B-Instruct --host 0.0.0.0 --port 8000

测试:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{

"model": "/home/llmlocal/qwen/qwen/Qwen2-7B-Instruct",

"messages": [

{"role": "system", "content": "You are a helpful assistant."},

{"role": "user", "content": "Tell me something about large language models."}

],

"temperature": 0.7,

"top_p": 0.8,

"repetition_penalty": 1.05,

"max_tokens": 512

}'

更改ragflow的MODEL_NAME = "/home/llmlocal/qwen/qwen/Qwen2-7B-Instruct" 路径在rag里的chat_model

相关推荐
紫雾凌寒29 分钟前
计算机视觉 |解锁视频理解三剑客——TimeSformer
python·深度学习·神经网络·计算机视觉·transformer·timesformer
程序员杰哥1 小时前
测试用例详解
自动化测试·软件测试·python·功能测试·测试工具·职场和发展·测试用例
go54631584657 小时前
本地部署 GitHub 上的 Python 人脸识别项目
开发语言·python·github
FreakStudio8 小时前
手把手教你用 MicroPython 玩转幻尔串口舵机,代码+教程全公开
python·嵌入式·大学生·面向对象·技术栈·电子diy·电子计算机
tekin8 小时前
基于 Python 开发在线多人游戏服务器案例解析
服务器·python·游戏·在线多人游戏服务器
让学习成为一种生活方式10 小时前
libGL.so.1: cannot open shared object file: No such file or directory-linux022
linux·开发语言·python
java1234_小锋10 小时前
一周学会Flask3 Python Web开发-Jinja2模板继承和include标签使用
python·flask·flask3
图书馆钉子户10 小时前
from flask_session import Session 为什么是Session(app)这么用?
python·flask·mybatis
一只蜗牛儿10 小时前
Sherpa-ONNX:说话人识别与语音识别自动开启(VAD)+ Python API 完整指南
人工智能·python·语音识别
结衣结衣.10 小时前
【OpenCV】入门教学
图像处理·人工智能·python·opencv