#LatentSync技术原理

字节跳动开源的基于音频条件潜在扩散模型的端到端唇同步框架,基于潜在扩散模型,以音频条件潜在扩散模型为基础，利用 Stable Diffusion 强大能力，直接建模复杂的音频与视觉之间的关系，实现高质量的唇形同步. 从而制作虚拟主播与数字人等应用

#部署环境

win11 16G显存 (4060Ti)

git version 2.47.1.windows.2

conda 24.11.3

#前期准备

cuda环境

检查:

nvcc --version

nvidia-smi

#部署流程

下载项目源码:

git clone https://github.com/bytedance/LatentSync.git

conda使用国内镜像:

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/

创建python环境:

conda create -n ltsenv python==3.10

进入隔离环境:

conda activate ltsenv

pyTorch官网:

https://pytorch.org/

选择CUDA 12.4 :

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

注意 windows 环境下, python依赖 triton 2.2版本无法使用

需要手动替换win支持的低版本手动安装whl包

安装视频库:

conda install -c conda-forge ffmpeg

windows :
Please check your internet connection. This can happen if your antivirus software blocks the download of this file. You can install manually by following these steps:

Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.3/frpc_windows_amd64.exe
Rename the downloaded file to: frpc_windows_amd64_v0.3
Move the file to this location: C:\DevTools\anaconda3\envs\ltsenv\lib\site-packages\gradio

以上步骤全部完成后需要手动下载以下文件: (因hugging face仓库国内访问受限)

https://hf-mirror.com/ByteDance/LatentSync/resolve/main/whisper/tiny.pt
https://hf-mirror.com/ByteDance/LatentSync/resolve/main/latentsync_syncnet.pt
https://hf-mirror.com/ByteDance/LatentSync/resolve/main/latentsync_unet.pt

或直接搜索国内镜像源 https://hf-mirror.com/ 镜像站 >搜索 LatentSync

完成后进入项目根目录

创建checkpoints / whisper 2个文件夹

结构如下:

./checkpoints/

|--latentsync_unet.pt

|--latentsync_syncnet.pt

|--whisper

--thiy.pt