深度学习系列76:流式tts的一个简单实现

1. 概述

使用queue,producer不断向queue中添加audio,然后consumer不断从queue中消费audio。

下面的样例使用melo来生成语音,需要先下载melo.tts。模型在https://myshell-public-repo-hosting.s3.amazonaws.com/openvoice/basespeakers/ZH/checkpoint.pth,config在https://myshell-public-repo-hosting.s3.amazonaws.com/openvoice/basespeakers/ZH/config.json。

2. 代码部分

复制代码
from melo import utils
from melo.models import SynthesizerTrn
from melo.split_utils import split_sentences_zh
from melo.download_utils import load_or_download_config, load_or_download_model
from queue import Queue
from threading import Thread
import numpy as np
from pydub import playback
import torch, audiosegment
device = 'mps'
hps = load_or_download_config('ZH', use_hf=True, config_path=None)
num_languages = hps.num_languages
num_tones = hps.num_tones
symbols = hps.symbols
model = SynthesizerTrn(len(symbols),hps.data.filter_length // 2 + 1,hps.train.segment_size // hps.data.hop_length,n_speakers=hps.data.n_speakers,num_tones=num_tones,num_languages=num_languages,**hps.model,).to(device)
model.eval()
symbol_to_id = {s: i for i, s in enumerate(symbols)}
checkpoint_dict = load_or_download_model('ZH', device, use_hf=True, ckpt_path=None)
model.load_state_dict(checkpoint_dict['model'], strict=True)
def play(text):
    texts = text.split('。')
    speaker_id = 1
    def producer(queue):
        for i,t in enumerate(texts):
            bert, ja_bert, phones, tones, lang_ids = utils.get_text_for_tts_infer(t, 'ZH_MIX_EN', hps, device, symbol_to_id)
            with torch.no_grad():
                x_tst = phones.to(device).unsqueeze(0)
                tones = tones.to(device).unsqueeze(0)
                lang_ids = lang_ids.to(device).unsqueeze(0)
                bert = bert.to(device).unsqueeze(0)
                ja_bert = ja_bert.to(device).unsqueeze(0)
                x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
                del phones
                speakers = torch.LongTensor([speaker_id]).to(device)
                audio = model.infer(x_tst,x_tst_lengths,speakers,tones,lang_ids,bert,ja_bert,
                        sdp_ratio=0,noise_scale=0,noise_scale_w=0,length_scale=1,)[0][0, 0].data.cpu().float().numpy()
                del x_tst, tones, lang_ids, bert, ja_bert, x_tst_lengths, speakers
            queue.put(audio)
    
    def consumer(queue):
        while True:
            audio = queue.get()    
            playback.play(audiosegment.from_numpy_array(audio.astype(np.float32),hps.data.sampling_rate))
            queue.task_done()
            
    q = Queue()
    q.put(np.zeros(int(hps.data.sampling_rate * 0.1)))
    t1 = Thread(target = producer, args=(q,))
    t2 = Thread(target=consumer, args=(q,))
    t2.daemon = True  # 线程2是无限循环需要设置守护线程以便主线程退出
    
    t1.start()
    t2.start()
    
    t1.join()  # 等待所有项被生产
    q.join()  # 等待所有项被消费

text = "..."
play(text)
相关推荐
亲持红叶几秒前
GLU 变种:ReGLU 、 GEGLU 、 SwiGLU
人工智能·深度学习·神经网络·激活函数
说私域几秒前
线上协同办公时代:以开源AI大模型等工具培养网感,拥抱职业变革
人工智能·开源
群联云防护小杜2 分钟前
深度隐匿源IP:高防+群联AI云防护防绕过实战
运维·服务器·前端·网络·人工智能·网络协议·tcp/ip
摘星编程8 分钟前
构建智能客服Agent:从需求分析到生产部署
人工智能·需求分析·智能客服·agent开发·生产部署
不爱学习的YY酱11 分钟前
信息检索革命:Perplexica+cpolar打造你的专属智能搜索中枢
人工智能
whaosoft-1431 小时前
51c自动驾驶~合集7
人工智能
刘晓倩4 小时前
Coze智能体开发实战-多Agent综合实战
人工智能·coze
石迹耿千秋5 小时前
迁移学习--基于torchvision中VGG16模型的实战
人工智能·pytorch·机器学习·迁移学习
路人蛃8 小时前
通过国内扣子(Coze)搭建智能体并接入discord机器人
人工智能·python·ubuntu·ai·aigc·个人开发
CV-杨帆9 小时前
论文阅读:arxiv 2025 A Survey of Large Language Model Agents for Question Answering
论文阅读·人工智能·语言模型