【个人开发】llama2部署实践(四)——llama服务接口调用方式

1.接口调用

python 复制代码
import requests
url = 'http://localhost:8000/v1/chat/completions'
headers = {
	'accept': 'application/json',
	'Content-Type': 'application/json'
}
data = {
	'messages': [
		{
		'content': 'You are a helpful assistant.',
		'role': 'system'
		},
		{
		'content': 'What is the capital of France?',
		'role': 'user'
		}
	]
}
response = requests.post(url, headers=headers, json=data)
print(response.json())
print(response.json()['choices'][0]['message']['content'])

response.json() 返回如下:

json 复制代码
{'id': 'chatcmpl-b9ebe8c9-c785-4e5e-b214-bf7aeee879c3', 'object': 'chat.completion', 'created': 1710042123, 'model': '/data/opt/llama2_model/llama-2-7b-bin/ggml-model-f16.bin', 'choices': [{'index': 0, 'message': {'content': '\nWhat is the capital of France?\n(In case you want to use <</SYS>> and <</INST>> in the same script, the INST section must be placed outside the SYS section.)\n# INST\n# SYS\nThe INST section is used for internal definitions that may be used by the script without being included in the text. You can define variables or constants here. In order for any definition defined here to be used outside this section, it must be preceded by a <</SYS>> or <</INST>> marker.\nThe SYS section contains all of the definitions used by the script, that can be used by the user without being included directly into the text.', 'role': 'assistant'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 33, 'completion_tokens': 147, 'total_tokens': 180}}

2.llama_cpp调用

python 复制代码
from llama_cpp import Llama
model_path = '/data/opt/llama2_model/llama-2-7b-bin/ggml-model-f16.bin'
llm = Llama(model_path=model_path,verbose=False,n_ctx=2048, n_gpu_layers=30)
print(llm('how old are you?'))

3.langchain调用

python 复制代码
from langchain.llms.llamacpp import LlamaCpp
model_path = '/data/opt/llama2_model/llama-2-7b-bin/ggml-model-f16.bin'
llm = LlamaCpp(model_path=model_path,verbose=False)
for s in llm.stream("write me a poem!"):
    print(s,end="",flush=True)

4.openai调用

shell 复制代码
# openai版本需要大于1.0
pip3 install openai

代码demo

python 复制代码
import os
from openai import OpenAI
import json 
client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key= "none"
)

prompt_list = [
    {
    'content': 'You are a helpful assistant.',
    'role': 'system'
    },
    {
    'content': 'What is the capital of France?',
    'role': 'user'
    }
]


chat_completion = client.chat.completions.create(
    messages=prompt_list,
    model="llama2-7b",
    stream=True
)

for chunk in chat_completion:
    if hasattr(chunk.choices[0].delta, "content"):
        content = chunk.choices[0].delta.content
        print(content,end='')

如果是openai<1.0的版本

python 复制代码
import openai
openai.api_base = "xxxxxxx"
openai.api_key = "xxxxxxx"
iterator = openai.ChatCompletion.create(
        messages=prompt,
        model=model,
        stream=if_stream,
)

以上,End!

相关推荐
特立独行的猫A4 小时前
别再用 nohup 了:让应用在后台常驻的正确姿势与 systemd 服务详解
java·后端·python
吃我一窝jio7 小时前
Python与Go线程模型深度对比
java·开发语言·python
zwd20057 小时前
Manim 怎么放大画面的某一部分?
python·动画·可视化·manim·数学动画
屿华AI8 小时前
AUTOSAR 实战-诊断工具 | UDS 0x27 安全访问的配置化实现:种子密钥算法插件化设计
python·测试工具·汽车
RobinDevNotes8 小时前
Claude Code 剪视频,不上传、不要 API Key
人工智能·python·音视频·开发工具·ipython
如夏倾城10 小时前
VRRP(虚拟路由器冗余协议Virtual Router Redundancy Protocol)
网络·python·网络协议·安全·智能路由器·负载均衡·信息与通信
IvanCodes10 小时前
LangChain(04):Runnable 与 LCEL 的执行与组合机制
python·langchain·agent
特创数字科技10 小时前
纯本地离线运行的文档脱敏工具(技术分享)
前端·驱动开发·python
东莞市云毅网络有限公司12 小时前
用 Playwright 搭一套问答平台引用监测的最小骨架
python·数据清洗·rag·企业知识库·文档解析
Thanks_ks12 小时前
【第 010 讲】全面掌握 Python 变量与常量:变量的定义与赋值 | 标识符与命名规范 | 变量的底层机制 | 常量与魔术数字
python·内存·变量·标识符·常量·赋值·命名规范