SenseVoice微调

1. 环境安装

需要同时安装Sensevoice和FunASR，本文在root用户下操作

1. FunASR

   cd /home/ok/opt
   git clone https://github.com/modelscope/FunASR.git && cd FunASR
   pip3 install -e ./

2. SenseVoice-small

   cd /home/ok/opt
   git clone https://github.com/FunAudioLLM/SenseVoice.git

3. 下载模型

   modelscope download --model iic/SenseVoiceSmall（默认会下载到: /root/.cache/modelscope/hub/models/iic/SenseVoiceSmall）

2. 数据准备

1. 准备人工标注或者公开的数据集文件：train_wav.scp、train_text.txt，例如：

• 文件：train_wav.scp

  EZ8PX4HQSTTXJ6QF /home/ok/data/wav/1782442538723.wav
  VW3T6806N3564V0T /home/ok/data/wav/1782442528241.wav
  XS27SYMA6WCKXKV8 /home/ok/data/wav/1782441948724.wav
  M4Q2M93YMW8NK0LS /home/ok/data/wav/1782441923314.wav
  SI5BXYHY612QQDW6 /home/ok/data/wav/1782441909560.wav
  ...

• 文件：train_text.txt

  EZ8PX4HQSTTXJ6QF Okay, copy message.
  VW3T6806N3564V0T 你在哪里.
  XS27SYMA6WCKXKV8 我离你还有800.3米。
  M4Q2M93YMW8NK0LS Yes, correct. Yes, correct.
  SI5BXYHY612QQDW6 塔台呼叫0907
  ...

  把上面两个文件放到目录：/home/ok/data
  对应的音频文件放到目录：/home/ok/data/wav

2. 生成训练数据集和验证集文件: train.jsonl、val.jsonl

1. 使用命令生成 train.jsonl

   cd /home/ok/opt/SenseVoice
   
   sensevoice2jsonl \
   ++scp_file_list='["/home/ok/opt/data/train_wav.scp", "/home/ok/opt/data/train_text.txt"]' \
   ++data_type_list='["source", "target"]' \
   ++jsonl_file_out="/home/ok/opt/data/train.jsonl" \
   ++model_dir='iic/SenseVoiceSmall'

   命令会自动生成，完成后可以检查内容，确认text_language、emo_target等是否正确

2. 创建文件 /home/ok/opt/data/val.jsonl，并从train.jsonl中复制出20%到文件val.jsonl中。

3. 开始训练

1. 根据自己的机器配置修改脚本文件： /home/ok/opt/SenseVoice/finetune.sh。

   例如我的是4090D，修改如下：

   # Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
   #  MIT License  (https://opensource.org/licenses/MIT)
   
   workspace=`pwd`
   
   # which gpu to train or finetune
   export CUDA_VISIBLE_DEVICES="0"
   gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
   
   # model_name from model_hub, or model_dir in local path
   
   ## option 1, download model automatically
   model_name_or_model_dir="iic/SenseVoiceSmall"
   
   ## option 2, download model by git
   #local_path_root=${workspace}/modelscope_models
   #mkdir -p ${local_path_root}/${model_name_or_model_dir}
   #git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}
   #model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}
   
   
   # data dir, which contains: train.json, val.json
   train_data=/home/ok/opt/data/train.jsonl
   val_data=/home/ok/opt/data/val.jsonl
   
   # exp output dir
   output_dir="./outputs"
   log_file="${output_dir}/log.txt"
   
   deepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json
   
   mkdir -p ${output_dir}
   echo "log_file: ${log_file}"
   
   DISTRIBUTED_ARGS="
       --nnodes ${WORLD_SIZE:-1} \
       --nproc_per_node $gpu_num \
       --node_rank ${RANK:-0} \
       --master_addr ${MASTER_ADDR:-127.0.0.1} \
       --master_port ${MASTER_PORT:-26669}
   "
   
   echo $DISTRIBUTED_ARGS
   
   # funasr trainer path
   train_tool="/home/ok/opt/FunASR/funasr/bin/train_ds.py"
   
   torchrun $DISTRIBUTED_ARGS \
   ${train_tool} \
   ++model="${model_name_or_model_dir}" \
   ++trust_remote_code=true \
   ++train_data_set_list="${train_data}" \
   ++valid_data_set_list="${val_data}" \
   ++dataset_conf.data_split_num=1 \
   ++dataset_conf.batch_sampler="BatchSampler" \
   ++dataset_conf.batch_size=20000  \
   ++dataset_conf.sort_size=3000 \
   ++dataset_conf.batch_type="token" \
   ++dataset_conf.num_workers=16 \
   ++train_conf.max_epoch=30 \
   ++train_conf.log_interval=10 \
   ++train_conf.resume=true \
   ++train_conf.validate_interval=20 \
   ++train_conf.save_checkpoint_interval=20 \
   ++train_conf.keep_nbest_models=20 \
   ++train_conf.avg_nbest_model=10 \
   ++train_conf.use_deepspeed=false \
   ++train_conf.deepspeed_config=${deepspeed_config} \
   ++optim_conf.lr=0.0002 \
   ++train_conf.grad_clip=5.0 \
   ++output_dir="${output_dir}" &> ${log_file}

2. 开始训练，使用命令：

   cd /home/ok/opt/SenseVoice
   bash finetune.sh

3. 实时查看GPU使用情况：nvidia-smi -l

   

4. 实时查看训练情况：tail -f ./SenseVoice/outputs/log.txt | grep "loss_avg_rank"

   

4. 模型使用

训练结束后会在目录 /home/ok/opt/SenseVoice/outputs，一般选择 model.pt.base，可以直接复制到模型目录：/root/.cache/modelscope/hub/models/iic/SenseVoiceSmall 里，替换成原来的 model.pt，即可使用。