从ChatGPT发布至今,确实所有的应用都值得用大模型重新做一遍。国内外对基底大模型卷了又卷,新生的应用也在模型的迭代过程中,起起伏伏。
但可以坚信的是,AGI的方向和每个时代人们永远在变的不变的需求。
而求外不如求己,生活中的点点滴滴,就是我这个小码农的产品经理,所以也决定开个系列,监督自己,整合下学到的东西给自己留个记录。
第一个应用灵感来源于钢铁侠的Jarvis,喊一声可以帮你做任何事。
例如,我有事情在忙,我没法操作手机和电脑,需要Jarvis帮我用微信哄一哄老婆先。
这时候Jarvis接收到我的信息,解析出操作步骤和内容,执行打开微信,搜索老婆,再输入"老婆,我等等就回来跪榴莲。"并发送,完毕。
当然,说不玄乎点就是一个超级智能体 + 多感知 + 多机互联。
根据上述Jarvis场景,我们拆解详细的技术方案:
- 智能体:我们有大模型,结合自己写plan,写workflow
- 多感知:做个电脑应用,支持键盘输入+语音输入
- 多机互联:自动抓取windows窗体,到特定模块实现交互(这里可能不止是多个系统终端、网页,也可以不断地加入其他新的终端交互形式,但我们暂时以操作windows窗体为目标)
完整项目☞:github.com/metaimagine... ⭐持续更新,欢迎Star⭐
下面我们开始第一部分吧!
语言:Python
ASR:Sherpa-onnx
LLM:Deepseek
Agent框架:Agently
一、 接入ASR
这里使用👉Sherpa-onnx,实测用cpu都跑得六六的。
因为我们语音交互,必不可少用到麦克风,我们直奔主题:
1. 安装依赖
bash
pip install sounddevice sherpa_onnx
2. 代码
其中,使用带节点检测的代码: github.com/k2-fsa/sher... 也可以直接用我改后的,新建文件microphone_asr.py
,填入:
python
#!/usr/bin/env python3
# Please refer to
# https://k2-fsa.github.io/sherpa/onnx/pretrained_models/online-paraformer/paraformer-models.html#
# to download pre-trained sherpa
import os.path
import queue
from typing import Generator
import sounddevice as sd
import sys
import logging
import sherpa_onnx
logger = logging.getLogger(__name__)
class AsrHandler:
def __init__(self, model_path, debug=False):
self.recognizer = None
self.sentence_q = queue.Queue()
self.init_recognizer(model_path)
self.debug = debug
def init_recognizer(self, model_path):
encoder = os.path.join(model_path, "encoder.int8.onnx")
decoder = os.path.join(model_path, "decoder.int8.onnx")
tokens = os.path.join(model_path, "tokens.txt")
self.recognizer = sherpa_onnx.OnlineRecognizer.from_paraformer(
tokens=tokens,
encoder=encoder,
decoder=decoder,
num_threads=2,
sample_rate=16000,
feature_dim=80,
enable_endpoint_detection=True,
rule1_min_trailing_silence=2.4,
rule2_min_trailing_silence=1.2,
rule3_min_utterance_length=300, # it essentially disables this rule
)
@staticmethod
def show_devices():
devices = sd.query_devices()
if len(devices) == 0:
logger.info("No microphone devices found")
sys.exit(0)
logger.info(devices)
default_input_device_idx = sd.default.device[0]
logger.info(f'Use default device: {devices[default_input_device_idx]["name"]}')
def handle(self) -> Generator:
logger.info("Microphone Asr Started! Please speak...")
# The model is using 16 kHz, we use 48 kHz here to demonstrate that
# sherpa-onnx will do resampling inside.
sample_rate = 48000
samples_per_read = int(0.5 * sample_rate) # 0.1 second = 100 ms
stream = self.recognizer.create_stream()
last_result = ""
segment_id = 0
try:
with sd.InputStream(
channels=1, dtype="float32", samplerate=sample_rate
) as s:
while True:
samples, _ = s.read(samples_per_read) # a blocking read
samples = samples.reshape(-1)
stream.accept_waveform(sample_rate, samples)
while self.recognizer.is_ready(stream):
self.recognizer.decode_stream(stream)
is_endpoint = self.recognizer.is_endpoint(stream)
result = self.recognizer.get_result(stream)
if result and (last_result != result):
last_result = result
if self.debug: logger.info("\r{}:{}".format(segment_id, result))
if is_endpoint:
if result:
if self.debug:logger.info("\r{}:{}".format(segment_id, result))
segment_id += 1
# generator result
yield result
self.recognizer.reset(stream)
except sd.PortAudioError as e:
logger.exception(f"no input device found: {e}")
if __name__ == "__main__":
try:
asr_gen = AsrHandler(model_path="YOUR_MODEL_DIR_PATH").handle()
for chunk in asr_gen:
print("ASR Result : ", chunk)
except KeyboardInterrupt:
print("\nCaught Ctrl + C. Exiting")
3. 模型下载
模型下载指导: k2-fsa.github.io/sherpa/onnx... 实际上,就是里边wget的地址,我直接给其中一个模型地址的: hub.nuaa.cf/k2-fsa/sher... 下载后需要双解压(.tar.bz2),不清楚的可以问LLM。
4. 运行
解压后,把路径填到上述文件的YOUR_MODEL_DIR_PATH
中,
运行python microphone_asr.py
代码即可。
至此,ASR部分大功告成。
二、基于Agently接入大模型
这一部分讲解如何结合LLM + ASR,实现流畅交互。
接入LLM免不了后续的管理,如workflow、长短期记忆等等,那就需要引入个Agent框架,可选方案有llama-index、langchain、Agently。而从易用性、可视化上,后者Agently则比较契合我们,主打一个上手就用。
1. 安装
bash
pip install -U Agently
2. 调用大模型
创建agent.py
设置环境变量API_KEY \ API_URL \ MODEL
,加入以下代码运行之:
python
import os
import Agently
from dotenv import load_dotenv
load_dotenv()
agent_factory = Agently.AgentFactory()
agent_factory \
.set_settings("current_model", "OpenAI") \
.set_settings("model.OpenAI.auth", {"api_key": os.environ["API_KEY"]}) \
.set_settings("model.OpenAI.url", os.environ["API_URL"]) \
.set_settings("model.OpenAI.options", {"model": os.environ["MODEL"]})
agent = agent_factory.create_agent("agent_id_1", is_debug=True)
agent \
.input("Hello, how are you?") \
.start()
显示以下内容,说明调试成功:
关联ASR和LLM
我们需要实现个简单的 [ 麦克风说法 - 大模型显示回复 ] 的流程,就需要将asr从语音转到的文字,作为问题传给LLM大模型,让大模型回复。
在ASR部分我们知道,ASR输出的是个生成器Generator,生成的内容是单句检测出来的文本,调用方法为:
python
asr_gen = AsrHandler(model_path="YOUR_MODEL_DIR_PATH").handle()
for sentence in asr_gen:
process(sentence)
那么,对应LLM在Agently框架下,即为:
python
for sentence in asr_gen:
agent \
.input(sentence) \
.start()
整合后的代码如下:
python
import os
import Agently
from dotenv import load_dotenv
load_dotenv()
#!/usr/bin/env python3
# Please refer to
# https://k2-fsa.github.io/sherpa/onnx/pretrained_models/online-paraformer/paraformer-models.html#
# to download pre-trained sherpa
import os.path
import queue
from typing import Generator
import sounddevice as sd
import sys
import logging
import sherpa_onnx
logger = logging.getLogger(__name__)
class AsrHandler:
def __init__(self, model_path, debug=False):
self.recognizer = None
self.sentence_q = queue.Queue()
self.init_recognizer(model_path)
self.debug = debug
def init_recognizer(self, model_path):
encoder = os.path.join(model_path, "encoder.int8.onnx")
decoder = os.path.join(model_path, "decoder.int8.onnx")
tokens = os.path.join(model_path, "tokens.txt")
self.recognizer = sherpa_onnx.OnlineRecognizer.from_paraformer(
tokens=tokens,
encoder=encoder,
decoder=decoder,
num_threads=2,
sample_rate=16000,
feature_dim=80,
enable_endpoint_detection=True,
rule1_min_trailing_silence=2.4,
rule2_min_trailing_silence=1.2,
rule3_min_utterance_length=300, # it essentially disables this rule
)
@staticmethod
def show_devices():
devices = sd.query_devices()
if len(devices) == 0:
logger.info("No microphone devices found")
sys.exit(0)
logger.info(devices)
default_input_device_idx = sd.default.device[0]
logger.info(f'Use default device: {devices[default_input_device_idx]["name"]}')
def handle(self) -> Generator:
logger.info("Microphone Asr Started! Please speak...")
# The model is using 16 kHz, we use 48 kHz here to demonstrate that
# sherpa-onnx will do resampling inside.
sample_rate = 48000
samples_per_read = int(0.5 * sample_rate) # 0.1 second = 100 ms
stream = self.recognizer.create_stream()
last_result = ""
segment_id = 0
try:
with sd.InputStream(
channels=1, dtype="float32", samplerate=sample_rate
) as s:
while True:
samples, _ = s.read(samples_per_read) # a blocking read
samples = samples.reshape(-1)
stream.accept_waveform(sample_rate, samples)
while self.recognizer.is_ready(stream):
self.recognizer.decode_stream(stream)
is_endpoint = self.recognizer.is_endpoint(stream)
result = self.recognizer.get_result(stream)
if result and (last_result != result):
last_result = result
if self.debug: logger.info("\r{}:{}".format(segment_id, result))
if is_endpoint:
if result:
if self.debug:logger.info("\r{}:{}".format(segment_id, result))
segment_id += 1
# generator result
yield result
self.recognizer.reset(stream)
except sd.PortAudioError as e:
logger.exception(f"no input device found: {e}")
if __name__ == "__main__":
agent_factory = Agently.AgentFactory()
agent_factory \
.set_settings("current_model", "OpenAI") \
.set_settings("model.OpenAI.auth", {"api_key": os.environ["API_KEY"]}) \
.set_settings("model.OpenAI.url", os.environ["API_URL"]) \
.set_settings("model.OpenAI.options", {"model": os.environ["MODEL"]})
agent = agent_factory.create_agent("agent_id_1", is_debug=True)
try:
asr_gen = AsrHandler(model_path=".models/asr/sherpa/sherpa-onnx-streaming-paraformer-bilingual-zh-en").handle()
for sentence in asr_gen:
agent \
.input(sentence) \
.start()
except KeyboardInterrupt:
print("\nCaught Ctrl + C. Exiting")
三、结束
本文讲述了实现windows上Jarvis的简单技术方案,同时完成了ASR和LLM的接入,实现了二者的丝滑交互。
下一篇将会进入交互模块,如何展示Agent的强大功能,如何实现窗体交互。
感谢观看,请不吝三连~
完整项目☞:github.com/metaimagine... ⭐持续更新,欢迎Star⭐
我是🫣MetaImagine,愿未来我们都有元想象能力。