LLM之基于llama-index部署本地embedding与GLM-4模型并初步搭建RAG（其他大模型也可，附上ollma方式运行）

黑夜寻白天2024-08-23 22:40

前言

日常没空，留着以后写

llama-index简介

官网：https://docs.llamaindex.ai/en/stable/

简介也没空，以后再写

注：先说明，随着官方的变动，代码也可能变动，大家运行不起来，可以进官网查查资料

加载本地embedding模型

如果没有找到 llama_index.embeddings.huggingface

那么：pip install llama_index-embeddings-huggingface

还不行进入官网，输入huggingface进行搜索

python 复制代码 
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import Settings

Settings.embed_model = HuggingFaceEmbedding(
    model_name=f"{embed_model_path}",device='cuda'

)

加载本地LLM模型

还是那句话，如果以下代码不行，进官网搜索Custom LLM Model

python 复制代码 
from llama_index.core.llms import (
    CustomLLM,
    CompletionResponse,
    CompletionResponseGen,
    LLMMetadata,
)
from llama_index.core.llms.callbacks import llm_completion_callback
from transformers import AutoTokenizer, AutoModelForCausalLM

class GLMCustomLLM(CustomLLM):
    context_window: int = 8192  # 上下文窗口大小
    num_output: int = 8000  # 输出的token数量
    model_name: str = "glm-4-9b-chat"  # 模型名称
    tokenizer: object = None  # 分词器
    model: object = None  # 模型
    dummy_response: str = "My response"

    def __init__(self, pretrained_model_name_or_path):
        super().__init__()

        # GPU方式加载模型
        self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, device_map="cuda", trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, device_map="cuda", trust_remote_code=True).eval()

        # CPU方式加载模型
        # self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, device_map="cpu", trust_remote_code=True)
        # self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, device_map="cpu", trust_remote_code=True)
        self.model = self.model.float()

    @property
    def metadata(self) -> LLMMetadata:
        """Get LLM metadata."""
        # 得到LLM的元数据
        return LLMMetadata(
            context_window=self.context_window,
            num_output=self.num_output,
            model_name=self.model_name,
        )

    # @llm_completion_callback()
    # def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
    #     return CompletionResponse(text=self.dummy_response)
    #
    # @llm_completion_callback()
    # def stream_complete(
    #     self, prompt: str, **kwargs: Any
    # ) -> CompletionResponseGen:
    #     response = ""
    #     for token in self.dummy_response:
    #         response += token
    #         yield CompletionResponse(text=response, delta=token)

    @llm_completion_callback()  # 回调函数
    def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
        # 完成函数
        print("完成函数")

        inputs = self.tokenizer.encode(prompt, return_tensors='pt').cuda()  # GPU方式
        # inputs = self.tokenizer.encode(prompt, return_tensors='pt')  # CPU方式
        outputs = self.model.generate(inputs, max_length=self.num_output)
        response = self.tokenizer.decode(outputs[0])
        return CompletionResponse(text=response)

    @llm_completion_callback()
    def stream_complete(
        self, prompt: str, **kwargs: Any
    ) -> CompletionResponseGen:
        # 流式完成函数
        print("流式完成函数")

        inputs = self.tokenizer.encode(prompt, return_tensors='pt').cuda()  # GPU方式
        # inputs = self.tokenizer.encode(prompt, return_tensors='pt')  # CPU方式
        outputs = self.model.generate(inputs, max_length=self.num_output)
        response = self.tokenizer.decode(outputs[0])
        for token in response:
            yield CompletionResponse(text=token, delta=token)

基于本地模型搭建简易RAG

python 复制代码 
from typing import Any

from llama_index.core.llms import (
    CustomLLM,
    CompletionResponse,
    CompletionResponseGen,
    LLMMetadata,
)
from llama_index.core.llms.callbacks import llm_completion_callback
from transformers import AutoTokenizer, AutoModelForCausalLM
from llama_index.core import Settings,VectorStoreIndex,SimpleDirectoryReader
from llama_index.embeddings.huggingface import HuggingFaceEmbedding


class GLMCustomLLM(CustomLLM):
    context_window: int = 8192  # 上下文窗口大小
    num_output: int = 8000  # 输出的token数量
    model_name: str = "glm-4-9b-chat"  # 模型名称
    tokenizer: object = None  # 分词器
    model: object = None  # 模型
    dummy_response: str = "My response"

    def __init__(self, pretrained_model_name_or_path):
        super().__init__()

        # GPU方式加载模型
        self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, device_map="cuda", trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, device_map="cuda", trust_remote_code=True).eval()

        # CPU方式加载模型
        # self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, device_map="cpu", trust_remote_code=True)
        # self.model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, device_map="cpu", trust_remote_code=True)
        self.model = self.model.float()

    @property
    def metadata(self) -> LLMMetadata:
        """Get LLM metadata."""
        # 得到LLM的元数据
        return LLMMetadata(
            context_window=self.context_window,
            num_output=self.num_output,
            model_name=self.model_name,
        )

    # @llm_completion_callback()
    # def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
    #     return CompletionResponse(text=self.dummy_response)
    #
    # @llm_completion_callback()
    # def stream_complete(
    #     self, prompt: str, **kwargs: Any
    # ) -> CompletionResponseGen:
    #     response = ""
    #     for token in self.dummy_response:
    #         response += token
    #         yield CompletionResponse(text=response, delta=token)

    @llm_completion_callback()  # 回调函数
    def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
        # 完成函数
        print("完成函数")

        inputs = self.tokenizer.encode(prompt, return_tensors='pt').cuda()  # GPU方式
        # inputs = self.tokenizer.encode(prompt, return_tensors='pt')  # CPU方式
        outputs = self.model.generate(inputs, max_length=self.num_output)
        response = self.tokenizer.decode(outputs[0])
        return CompletionResponse(text=response)

    @llm_completion_callback()
    def stream_complete(
        self, prompt: str, **kwargs: Any
    ) -> CompletionResponseGen:
        # 流式完成函数
        print("流式完成函数")

        inputs = self.tokenizer.encode(prompt, return_tensors='pt').cuda()  # GPU方式
        # inputs = self.tokenizer.encode(prompt, return_tensors='pt')  # CPU方式
        outputs = self.model.generate(inputs, max_length=self.num_output)
        response = self.tokenizer.decode(outputs[0])
        for token in response:
            yield CompletionResponse(text=token, delta=token)


if __name__ == "__main__":


    # 定义你的LLM
    pretrained_model_name_or_path = r'/home/nlp/model/LLM/THUDM/glm-4-9b-chat'
    embed_model_path = '/home/nlp/model/Embedding/BAAI/bge-m3'

    Settings.embed_model = HuggingFaceEmbedding(
        model_name=f"{embed_model_path}",device='cuda'

    )

    Settings.llm = GLMCustomLLM(pretrained_model_name_or_path)

    documents = SimpleDirectoryReader(input_dir="home/xxxx/input").load_data()
    index = VectorStoreIndex.from_documents(
        documents,
    )


    # 查询和打印结果
    query_engine = index.as_query_engine()
    response = query_engine.query("萧炎的表妹是谁?")

    print(response)

ollama

python 复制代码 
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.ollama import Ollama

documents = SimpleDirectoryReader("data").load_data()

# bge-base embedding model
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")

# ollama
Settings.llm = Ollama(model="llama3", request_timeout=360.0)

index = VectorStoreIndex.from_documents(
    documents,
)

欢迎大家点赞或收藏

大家的点赞或收藏可以鼓励作者加快更新哟~

参加链接：

LlamaIndex中的CustomLLM（本地加载模型）
llamaIndex 基于GPU加载本地embedding模型

官网文档

官网_starter_example_loca

官网_usage_custom

相关推荐
小程故事多_8014 小时前
边缘端OCR+RAG文档智能问答系统，落地实践与全场景解析
ocr·rag
染指11101 天前
61.RAG-RAG存在的问题
人工智能·llama·rag·llama_index·llamaindex
独码侠1 天前
Dify 开源深度实践:从企业部署到二次开发的全景拆解
知识库·dify·rag·ai工作流
Esaka_Forever2 天前
RAG、提示工程、微调三大方案场景完整梳理与补充总结
rag
要开心吖ZSH4 天前
AI医疗分诊与健康咨询助手agent开发——（4-2）从零到一：我给AI医疗分诊助手加了个“知识库大脑“，终于搞懂了RAG是什么
java·docker·agent·医疗·rag
sugar__salt4 天前
Document 切割：RAG 知识库数据预处理实战
javascript·langchain·embedding·js·rag·cheerio
@atweiwei4 天前
Langchainrust:中LLM-as-a-Judge,用 Rust 实现一套 LLM 评估系统
开发语言·后端·ai·rust·llm·rag
呆呆敲代码的小Y5 天前
RAG 表格解析完全指南：xParse、PaddleOCR、MinerU 实测对比并接入Agent使用
ai·llm·agent·rag·表格解析·xparse
染指11106 天前
56.llama_index-查询引擎
人工智能·llama·rag·llama_index·llamaindex
53AI6 天前
RAG分块策略深度对比：主流方法的检索准确率差异与选型建议
知识库·rag·企业知识库·ai知识库·rag知识库
热门推荐
012026年7月AI圈大地震：GPT-5.6被政府限制、Claude入驻Slack、Anthropic自研芯片02GitHub 镜像站点03如何新建文件夹？ 电脑新建文件夹的4种方法04AI科技热点日报 | 2026年07月01日05幻兽帕鲁 - 服务器管理员权限与 GM 命令完全指南06国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）07AI 编程 IDE 全景解析 2026：Agent 全面接管开发链路082026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？092026 年 AI 大模型 & AI 编程工具实战全总结102026年AI技术突破与产业落地全景：从GPT-5到多模态智能体的新纪元