基于【PaddleMIX】的音乐快速生成

一、基于【PaddleMIX】的音乐快速生成

1.PaddleMIX简介

PaddleMIX是基于飞桨的跨模态大模型开发套件，聚合图像、文本、视频等多种模态，覆盖视觉语言预训练，文生图，文生视频等丰富的跨模态任务。提供开箱即用的开发体验，同时满足开发者灵活定制需求，探索通用人工智能。

2.地址

github.com/PaddlePaddl...

二、环境准备

1.PaddleMIX 安装

# 为快速可以用gitee做镜像
# !git clone https://github.com/PaddlePaddle/PaddleMIX.git --depth=1
!git clone https://gitee.com/livingbody/PaddleMIX.git --depth=1

%%capture
# 安装PaddleMIX
%cd ~/PaddleMIX
!pip install -r requirements.txt --user  -i https://pypi.tuna.tsinghua.edu.cn/simple
!pip install -e . --user  -i https://pypi.tuna.tsinghua.edu.cn/simple

2.appflow安装

appflow是PaddleMIX提供的一键预测组件，可快速完成模型推理。

%%capture
# 安装AppFlow推理
%cd ~/PaddleMIX
!pip install -r paddlemix/appflow/requirements.txt --user -i https://pypi.tuna.tsinghua.edu.cn/simple

3.ppdiffusers安装

PPDiffusers 是一款支持多种模态（如文本图像跨模态、图像、语音）扩散模型（Diffusion Model）训练和推理的国产化工具箱，依托于PaddlePaddle框架和PaddleNLP自然语言处理开发库。

%%capture
#ppdiffusers 安装
%cd ppdiffusers
!pip install -r ./requirements.txt --user  -i https://pypi.tuna.tsinghua.edu.cn/simple
!pip install -e . --user  -i https://pypi.tuna.tsinghua.edu.cn/simple

注意安装完毕需要重启notebook

三、 nltk_data数据准备

%%capture
!unzip /home/aistudio/data/data198535/nltk_data.zip -d ~/

四、 文本-音频生成

 文本条件的音频生成（Text-to-Audio Generation）

# music generation
# 快速推理

import paddle
import os
from paddlemix.appflow import Appflow


import scipy

# Text to music
task = Appflow(app="music_generation", models=["cvssp/audioldm"])
prompt = "A classic cocktail lounge vibe with smooth jazz piano and a cool, relaxed atmosphere."
# 低质量、中等质量、低沉质量、噪音干扰、差和低质量、听不清的质量、低保真质量
negative_prompt = 'low quality, average quality, muffled quality, noise interference, poor and low-grade quality, inaudible quality, low-fidelity quality'  
# 长度
audio_length_in_s = 120
num_inference_steps = 20
output_path = "tmp.wav"
result = task(prompt=prompt, negative_prompt=negative_prompt, num_inference_steps=num_inference_steps, audio_length_in_s=audio_length_in_s, generator = paddle.Generator().manual_seed(120))['result']
scipy.io.wavfile.write(output_path, rate=16000, data=result)

Fetching 14 files: 100%|██████████| 14/14 [00:00<00:00, 15351.70it/s]
[2023-12-03 15:05:37,610] [    INFO] - Loading configuration file /home/aistudio/.cache/paddlenlp/ppdiffusers/cvssp/audioldm/text_encoder/config.json
[2023-12-03 15:05:39,043] [    INFO] - Loading configuration file /home/aistudio/.cache/paddlenlp/ppdiffusers/cvssp/audioldm/vocoder/config.json


A classic cocktail lounge vibe with smooth jazz piano and a cool, relaxed atmosphere.


  0%|          | 0/20 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/variable_index.py:292: UserWarning: 1-D Tensor will be treat as advanced indexing in future version. Currently, 1-D Tensor means a scalar, not vector, and please modify it to 0-D Tensor. If advanced indexing is needed, please use `export FLAGS_set_to_1d=False` to set the flag.
  warnings.warn(
/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/variable_index.py:591: UserWarning: Warning: In Tensor '__getitem__', if the number of scalar elements in the index is equal to the rank of the Tensor, the output should be 0-D. In order to be consistent with the behavior of previous versions, it will be processed to 1-D. But it is not correct and will be removed in release 2.6. If 1-D is still wanted, please modify the index element from scalar to slice (e.g. 'x[i]' => 'x[i:i+1]').
  warnings.warn(
100%|██████████| 20/20 [00:14<00:00,  1.35it/s]

五、播放

生成的音频保存于 tmp.wav 目录下，下载播放即可。

import IPython
IPython.display.Audio('tmp.wav')