本地部署语音识别框架FunAudioLLM——Fun-ASR-Nano-2512模型

文章目录

功能概要

采用阿里通义实验室发布的开源项目 FunAudioLLM,甚至有识别方言的选项。基于FunAudioLLM工具库部署Fun-ASR-Nano-2512语音识别模型,实现以下功能:

(1)GPU运算

(2)利用VAD自动判断何时断句进行检测

(3)能够手动选择收音设备

(4)将检测结果显示在GUI中,若停顿超过固定时间则文字换行

基本环境配置

windows环境下安装anaconda创建虚拟环境,基于python使用pycharm调试,详情参考的我的文章:

https://blog.csdn.net/sinat_30065705/article/details/163958783?fromshare=blogdetail\&sharetype=blogdetail\&sharerId=163958783\&sharerefer=PC\&sharesource=sinat_30065705\&sharefrom=from_link

(1)安装最新版的anaconda和pycharm

最好都用最新版的,互相兼容。

anaconda下载地址:

https://www.anaconda.com/download

pycharm下载地址:社区版和收费版合并了,收费到期自动变成社区版

https://www.jetbrains.com/pycharm/download/

(2)base环境下安装mamba

bash 复制代码
conda update conda
conda install mamba -n base -c conda-forge
mamba --version  # 验证安装成功

(3)创建虚拟环境并安装依赖包

其中ffmpeg 需要收到下载windows版本单独安装,conda和pip安装都不好使。https://www.ffmpeg.org/download.html

bash 复制代码
mamba create -n llm_api python=3.11 -y
mamba activate llm_api

# PyTorch GPU(CUDA 12.4)
mamba install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia -y

# Transformers生态
mamba install transformers accelerate peft datasets scikit-learn -c conda-forge -y

# bitsandbytes(必须用pip,conda版容易编译失败)
pip install bitsandbytes -i https://pypi.tuna.tsinghua.edu.cn/simple

# Web服务 + 通信 + 工具
pip install fastapi uvicorn pyserial requests matplotlib ollama -i https://pypi.tuna.tsinghua.edu.cn/simple

# 语音识别(Whisper 依赖 torchaudio 已装)
pip install openai-whisper -i https://pypi.tuna.tsinghua.edu.cn/simple
# 我选择的是国产的Fun-ASR模型
pip install funasr[all] -i https://pypi.tuna.tsinghua.edu.cn/simple
# 若报错清理缓存后再试即可
mamba install -c ffmpeg -y
# 安装 modelscope 库
pip install modelscope
# 下载 Fun-ASR-Nano-2512 模型,默认下载到 C:\Users\Administrator\.cache\modelscope\models\FunAudioLLM--Fun-ASR-Nano-2512
modelscope download --model FunAudioLLM/Fun-ASR-Nano-2512

Fun-ASR-Nano-2512 模型下载

通过modelscope下载,下载后将模型文件转移至工程目录下,目录结构按照以下配置:

E:\Project\Fun_ASR\models\FunAudioLLM--Fun-ASR-Nano-2512

模型官方页面:https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512

modelscope下载代码:

bash 复制代码
modelscope download --model FunAudioLLM/Fun-ASR-Nano-2512

完整可运行代码

记得运行过程中要根据列出的音频设备信息输入数字指定收音设备

python 复制代码
import os
import time
import threading
import queue
import tempfile
import soundfile as sf
import pyaudio
import numpy as np
import tkinter as tk
from tkinter import scrolledtext
from funasr import AutoModel

# ==================== 用户配置区域 ====================
# 模型文件夹E:\Project\Fun_AS这个目录再往下就是models文件夹
os.environ["MODELSCOPE_CACHE"] = r"E:\Project\Fun_ASR"

# 指定热点词汇以校准常用词汇的语音识别
HOTWORDS = "王桑:200 吴桑:200 小傻瓜 龟儿子:90 中国人 地球人"
CONFIDENCE_THRESHOLD = 0.2 # 置信度控制
SILENCE_TIMEOUT_GUI = 3.0 # 换行时间间隔

VAD_FRAME_DURATION = 0.3 # vad每0.3秒自动判断是否有语音
SILENCE_DURATION = 0.6 # 静音0.3秒视为一段话结束

# ==================== 音频参数 ====================
CHUNK = int(16000 * 0.1)
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000

VAD_FRAME_SAMPLES = int(RATE * VAD_FRAME_DURATION)
SILENCE_FRAMES = int(SILENCE_DURATION / VAD_FRAME_DURATION)

# ==================== 全局变量 ====================
result_queue = queue.Queue()
is_running = True
MIC_DEVICE_INDEX = None


# ==================== 加载模型 ====================
def load_models():
    print("正在加载 ASR 模型...")
    asr_model = AutoModel(
        model="FunAudioLLM/Fun-ASR-Nano-2512",
        device="cuda:0",
        trust_remote_code=True,
        disable_update=True,
    )
    print("ASR 模型加载完成!")

    print("正在加载 VAD 模型...")
    vad_model = AutoModel(
        model="fsmn-vad",
        device="cuda:0",
        trust_remote_code=True,
        disable_update=True,
    )
    print("VAD 模型加载完成!")
    return asr_model, vad_model


# ==================== 主线程 ====================
def audio_and_asr_worker(asr_model, vad_model, mic_index):
    global is_running

    p = pyaudio.PyAudio()
    stream = p.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=RATE,
        input=True,
        input_device_index=mic_index,
        frames_per_buffer=CHUNK,
    )

    print(f"🎤 音频线程启动,VAD帧长: {VAD_FRAME_DURATION}s, 静音阈值: {SILENCE_DURATION}s")

    speech_buffer = b''
    is_speaking = False
    silence_counter = 0
    frame_buffer = b''

    while is_running:
        try:
            data = stream.read(CHUNK, exception_on_overflow=False)
            frame_buffer += data

            if len(frame_buffer) >= VAD_FRAME_SAMPLES * 2:
                vad_data = frame_buffer[:VAD_FRAME_SAMPLES * 2]
                frame_buffer = frame_buffer[VAD_FRAME_SAMPLES * 2:]

                # VAD 检测(保存为临时文件)
                audio_int16 = np.frombuffer(vad_data, dtype=np.int16)
                audio_float = audio_int16.astype(np.float32) / 32768.0

                with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
                    tmp_path = tmp.name
                    sf.write(tmp_path, audio_float, RATE)

                try:
                    vad_res = vad_model.generate(input=[tmp_path], batch_size=1, disable_progress=True)
                    has_speech = False
                    if vad_res and len(vad_res) > 0:
                        for item in vad_res:
                            if 'value' in item and item['value']:
                                has_speech = True
                                break
                except Exception as e:
                    print(f"VAD 异常: {e}")
                    has_speech = False
                finally:
                    try:
                        os.unlink(tmp_path)
                    except:
                        pass

                # ---------- 状态机 ----------
                if has_speech:
                    silence_counter = 0
                    if not is_speaking:
                        is_speaking = True
                        speech_buffer = b''
                        print("🔊 [语音开始]")
                    speech_buffer += vad_data
                else:
                    if is_speaking:
                        silence_counter += 1
                        speech_buffer += vad_data
                        if silence_counter >= SILENCE_FRAMES:
                            is_speaking = False
                            speech_len = len(speech_buffer)
                            print(f"🔇 [语音结束],累积 {speech_len} bytes ({speech_len / (RATE * 2):.2f} 秒)")

                            if speech_buffer:
                                # ===== 修复:增强 ASR 调用日志 =====
                                print("📤 [1] 开始转换为 float32...")
                                audio_int16_full = np.frombuffer(speech_buffer, dtype=np.int16)
                                audio_float_full = audio_int16_full.astype(np.float32) / 32768.0
                                print(f"📤 [2] 转换完成,shape: {audio_float_full.shape}")

                                with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
                                    tmp_path = tmp.name
                                    sf.write(tmp_path, audio_float_full, RATE)
                                print(f"📤 [3] 临时文件已保存: {tmp_path}")

                                try:
                                    print("📤 [4] 调用 ASR 识别...")
                                    res = asr_model.generate(
                                        input=[tmp_path],
                                        batch_size=1,
                                        disable_progress=True,
                                        hotword=HOTWORDS,
                                        language="zh",
                                        use_itn=True,
                                        output_timestamp=True,
                                        beam_size=5,
                                        decoding_ctc_weight=0.3,
                                    )
                                    print(f"📤 [5] ASR 返回: {res}")

                                    if res and len(res) > 0:
                                        timestamps = res[0].get("timestamps", [])
                                        if timestamps:
                                            filtered_tokens = []
                                            for token_info in timestamps:
                                                score = token_info.get("score", 0)
                                                token = token_info.get("token", "")
                                                if score >= CONFIDENCE_THRESHOLD:
                                                    filtered_tokens.append(token)
                                            filtered_text = ''.join(filtered_tokens).strip()
                                            if filtered_text:
                                                print(f"✅ 识别结果: {filtered_text}")
                                                result_queue.put(filtered_text)
                                            else:
                                                print("⚠️ 过滤后为空")
                                        else:
                                            text = res[0].get("text", "").strip()
                                            if text:
                                                print(f"✅ 识别结果: {text}")
                                                result_queue.put(text)
                                            else:
                                                print("⚠️ text 字段为空")
                                    else:
                                        print("⚠️ ASR 返回空或格式异常")
                                except Exception as e:
                                    print(f"❌ ASR 识别异常: {e}")
                                    import traceback
                                    traceback.print_exc()
                                finally:
                                    try:
                                        os.unlink(tmp_path)
                                        print("📤 [6] 临时文件已删除")
                                    except Exception as e:
                                        print(f"删除临时文件失败: {e}")
                                speech_buffer = b''
        except Exception as e:
            print(f"❌ 主循环异常: {e}")
            import traceback
            traceback.print_exc()
            break

    stream.stop_stream()
    stream.close()
    p.terminate()
    print("音频线程已停止。")


# ==================== GUI ====================
class VoiceApp:
    def __init__(self, root):
        self.root = root
        root.title("🎤 王桑的实时语音识别 (调试版) - FunASR")
        root.geometry("800x400")

        self.text_area = scrolledtext.ScrolledText(
            root, wrap=tk.WORD, font=("微软雅黑", 14),
            bg="#f0f0f0", fg="#333333"
        )
        self.text_area.pack(padx=10, pady=10, fill=tk.BOTH, expand=True)
        self.text_area.insert(tk.END, "等待语音输入...")
        self.text_area.see(tk.END)

        status_text = f"🔊 调试模式 | 阈值: {CONFIDENCE_THRESHOLD}"
        self.status = tk.Label(root, text=status_text, font=("微软雅黑", 9), fg="#555")
        self.status.pack(pady=(0, 10))

        self.last_display_time = None
        self.root.protocol("WM_DELETE_WINDOW", self.on_closing)
        self.root.after(100, self.update_text)

    def update_text(self):
        now = time.time()
        texts = []
        while not result_queue.empty():
            try:
                texts.append(result_queue.get_nowait())
            except queue.Empty:
                break

        if texts:
            if self.last_display_time is None or (now - self.last_display_time) > SILENCE_TIMEOUT_GUI:
                self.text_area.insert(tk.END, "\n" + texts[0])
            else:
                self.text_area.insert(tk.END, " " + texts[0])

            for t in texts[1:]:
                self.text_area.insert(tk.END, " " + t)

            self.text_area.see(tk.END)
            self.last_display_time = now

        self.root.after(100, self.update_text)

    def on_closing(self):
        global is_running
        is_running = False
        self.root.destroy()


# ==================== 设备选择 ====================
def select_microphone():
    p = pyaudio.PyAudio()
    print("\n===== 可用的录音设备列表 =====")
    input_devices = []
    for i in range(p.get_device_count()):
        info = p.get_device_info_by_index(i)
        if info['maxInputChannels'] > 0:
            input_devices.append((i, info['name']))
            print(f"  [{i}] {info['name']} (输入通道: {info['maxInputChannels']})")
    p.terminate()

    if not input_devices:
        print("❌ 未找到任何录音设备")
        exit(1)

    while True:
        try:
            choice = input("\n请选择设备编号: ")
            idx = int(choice)
            if any(idx == dev[0] for dev in input_devices):
                print(f"✅ 已选择: {[dev[1] for dev in input_devices if dev[0] == idx][0]}")
                return idx
            else:
                print("⚠️ 编号无效")
        except ValueError:
            print("⚠️ 请输入数字")


# ==================== 主程序 ====================
if __name__ == "__main__":
    mic_idx = select_microphone()
    asr_model, vad_model = load_models()

    asr_thread = threading.Thread(target=audio_and_asr_worker, args=(asr_model, vad_model, mic_idx), daemon=True)
    asr_thread.start()

    root = tk.Tk()
    app = VoiceApp(root)
    root.mainloop()

    print("程序退出。")

语音识别效果视频

语音识别效果视频

相关推荐
Csvn4 小时前
同一个问题问 100 遍,钱就扣 100 次?精确缓存让它只扣一次(C02)
人工智能·aigc·agent
罗西的思考5 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(2)
人工智能·算法·机器学习
罗西的思考5 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(1)
人工智能·算法·机器学习
汤姆宝库6 小时前
BPE分词器的完整实现:字节对合并与词表构建
开发语言·人工智能·c#
岁月宁静6 小时前
从零拆解 AI 语音模拟面试系统
人工智能·python·agent
武子康6 小时前
H3-World 接入 Game Agent:谁来判断门真的打开了?
人工智能·算法·机器学习
可乐ea7 小时前
MCP 和 CLI 差 42 倍:一次 GitHub 任务的上下文成本实测
人工智能·github·工具调用·ai智能体·openrouter·mcp协议·上下文优化
kyriewen7 小时前
编辑弹窗点了取消,列表里的数据怎么还是变了?
前端·javascript·人工智能
泯泷7 小时前
Multica 真正打动我的,不是更多 Agent,而是一种新的团队协作方式
人工智能·llm·agent
电科_银尘7 小时前
【深度学习项目实战·22】灰度图像着色:用小型U-Net学习从灰度到RGB
人工智能·深度学习·学习