SenseVoice微调 - 技术栈

1. 环境安装

需要同时安装Sensevoice和FunASR，本文在root用户下操作

FunASR

复制代码

cd /home/ok/opt
git clone https://github.com/modelscope/FunASR.git && cd FunASR
pip3 install -e ./

SenseVoice-small

复制代码

cd /home/ok/opt
git clone https://github.com/FunAudioLLM/SenseVoice.git

下载模型

复制代码

modelscope download --model iic/SenseVoiceSmall（默认会下载到: /root/.cache/modelscope/hub/models/iic/SenseVoiceSmall）

2. 数据准备

1. 准备人工标注或者公开的数据集文件：train_wav.scp、train_text.txt，例如：

文件：train_wav.scp

复制代码

EZ8PX4HQSTTXJ6QF /home/ok/data/wav/1782442538723.wav
VW3T6806N3564V0T /home/ok/data/wav/1782442528241.wav
XS27SYMA6WCKXKV8 /home/ok/data/wav/1782441948724.wav
M4Q2M93YMW8NK0LS /home/ok/data/wav/1782441923314.wav
SI5BXYHY612QQDW6 /home/ok/data/wav/1782441909560.wav
...

文件：train_text.txt

复制代码

EZ8PX4HQSTTXJ6QF Okay, copy message.
VW3T6806N3564V0T 你在哪里.
XS27SYMA6WCKXKV8 我离你还有800.3米。
M4Q2M93YMW8NK0LS Yes, correct. Yes, correct.
SI5BXYHY612QQDW6 塔台呼叫0907
...

把上面两个文件放到目录：/home/ok/data
对应的音频文件放到目录：/home/ok/data/wav

2. 生成训练数据集和验证集文件: train.jsonl、val.jsonl

使用命令生成 train.jsonl

复制代码

cd /home/ok/opt/SenseVoice

sensevoice2jsonl \
++scp_file_list='["/home/ok/opt/data/train_wav.scp", "/home/ok/opt/data/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_out="/home/ok/opt/data/train.jsonl" \
++model_dir='iic/SenseVoiceSmall'

命令会自动生成，完成后可以检查内容，确认text_language、emo_target等是否正确

创建文件 /home/ok/opt/data/val.jsonl，并从train.jsonl中复制出20%到文件val.jsonl中。

3. 开始训练

根据自己的机器配置修改脚本文件： /home/ok/opt/SenseVoice/finetune.sh。

例如我的是4090D，修改如下：

复制代码

# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
#  MIT License  (https://opensource.org/licenses/MIT)

workspace=`pwd`

# which gpu to train or finetune
export CUDA_VISIBLE_DEVICES="0"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')

# model_name from model_hub, or model_dir in local path

## option 1, download model automatically
model_name_or_model_dir="iic/SenseVoiceSmall"

## option 2, download model by git
#local_path_root=${workspace}/modelscope_models
#mkdir -p ${local_path_root}/${model_name_or_model_dir}
#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}
#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}


# data dir, which contains: train.json, val.json
train_data=/home/ok/opt/data/train.jsonl
val_data=/home/ok/opt/data/val.jsonl

# exp output dir
output_dir="./outputs"
log_file="${output_dir}/log.txt"

deepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json

mkdir -p ${output_dir}
echo "log_file: ${log_file}"

DISTRIBUTED_ARGS="
    --nnodes ${WORLD_SIZE:-1} \
    --nproc_per_node $gpu_num \
    --node_rank ${RANK:-0} \
    --master_addr ${MASTER_ADDR:-127.0.0.1} \
    --master_port ${MASTER_PORT:-26669}
"

echo $DISTRIBUTED_ARGS

# funasr trainer path
train_tool="/home/ok/opt/FunASR/funasr/bin/train_ds.py"

torchrun $DISTRIBUTED_ARGS \
${train_tool} \
++model="${model_name_or_model_dir}" \
++trust_remote_code=true \
++train_data_set_list="${train_data}" \
++valid_data_set_list="${val_data}" \
++dataset_conf.data_split_num=1 \
++dataset_conf.batch_sampler="BatchSampler" \
++dataset_conf.batch_size=20000  \
++dataset_conf.sort_size=3000 \
++dataset_conf.batch_type="token" \
++dataset_conf.num_workers=16 \
++train_conf.max_epoch=30 \
++train_conf.log_interval=10 \
++train_conf.resume=true \
++train_conf.validate_interval=20 \
++train_conf.save_checkpoint_interval=20 \
++train_conf.keep_nbest_models=20 \
++train_conf.avg_nbest_model=10 \
++train_conf.use_deepspeed=false \
++train_conf.deepspeed_config=${deepspeed_config} \
++optim_conf.lr=0.0002 \
++train_conf.grad_clip=5.0 \
++output_dir="${output_dir}" &> ${log_file}

开始训练，使用命令：

复制代码

cd /home/ok/opt/SenseVoice
bash finetune.sh

实时查看GPU使用情况：nvidia-smi -l
实时查看训练情况：tail -f ./SenseVoice/outputs/log.txt | grep "loss_avg_rank"

4. 模型使用

训练结束后会在目录 /home/ok/opt/SenseVoice/outputs，一般选择 model.pt.base，可以直接复制到模型目录：/root/.cache/modelscope/hub/models/iic/SenseVoiceSmall 里，替换成原来的 model.pt，即可使用。