Paper : Wu J Z, Ge Y, Wang X, et al. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7623-7633.
Introduction : https://tuneavideo.github.io/
Code : https://github.com/showlab/Tune-A-Video
目录
- [一. 研究思路](#一. 研究思路)
- [二. 复现](#二. 复现)
-
- [1. man-skiing](#1. man-skiing)
- [2. man-spraying-fire](#2. man-spraying-fire)
一. 研究思路
为了充分利用 T2I 预训练大模型的强大生成能力，T2V 出现了一个新的范式 ------ One-Shot Video Tuning：使用一对文本 - 视频来训练 T2V 生成器，T2V 生成器需要从输入视频中捕获基本的运动信息，然后合成带有编辑文本语义的新视频。然而，T2I 生成的图像虽然满足文本的语义，但是在空间布局和具体内容上还是有比较大的差异。即使扩展了空间 self-attention，帧间还是有明显的不连续。
为了进一步学习连续的运动，本文提出了 Tune-A-Video，在时空维度上 (spatio-temporal dimension) 扩展 T2I 模型。然而，直接使用微调策略来更新参数会破坏 T2I 的先验知识，阻碍视频的生成。
因此，Tune-A-Video 引入了一种稀疏时空注意机制 (sparse spatio-temporal attention mechanism)，在视频编辑时只关注第一帧和前一帧；然后再对 T2I 模型进行微调，该过程中只更新 attention 块中的投影矩阵（即 Q、K、V）；推理时通过 DDIM 反演保持输入视频的结构，以尽可能保证相邻帧间的运动连续性。
二. 复现
- 平台：本地服务器
- 显卡：RTX 4090 24G
- 镜像：----
- 源码：https://github.com/showlab/Tune-A-Video
实验过程：
-
克隆仓库后，创建虚拟环境
tune_a_video并按照 README 安装依赖和下载权重，然后就可以开始训练； -
训练开始前，需要根据模型和训练数据路径修改
configs文件夹下的配置文件； -
训练时遇到报错：
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument 'find_unused_parameters=True' to 'torch.nn.parallel.DistributedDataParallel', and by making sure all 'forward' function outputs participate in calculating loss. If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's 'forward' function. Please include the loss function and the structure of the return value of 'forward' of your module when reporting this issue (e.g. list, dict, iterable). Parameter indices which did not receive grad for rank 0: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error.：
这是分布式数据并行引发的错误，最简单的办法就是只使用一个 GPU 进行训练 ^1^。具体的做法是，设置环境变量
CUDA_VISIBLE_DEVICE指定可用的 GPU：export CUDA_VISIBLE_DEVICES=0； -
随后又会遇到新的问题：
NotImplementedError: Using RTX 4000 series doesn't support faster communication broadband via P2P or IB. Please set 'NCCL_P2P_DISABLE="1"' and 'NCCL_IB_DISABLE="1" or use 'accelerate launch' which will do this automatically.：
这是因为 RTX 4000 系列显卡在某些情况下不支持通过 P2P 或 IB 来进行更快的通信，根据提示设置
NCCL_P2P_DISABLE和NCCL_IB_DISABLE环境变量即可：export NCCL_P2P_DISABLE=1; export NCCL_IB_DISABLE=1； -
解决以上问题后，就可以使用
accelerate launch train_tuneavideo.py --config="configs/man-skiing.yaml"指令进行训练。训练完成后，权重被写入outputs/XXX文件夹，配置文件中的 4 条验证结果被写入outputs/XXX/samples文件夹：
-
还可以使用 README 中提供的示例代码进行推理：
实验结果：
1. man-skiing
a man is skiing（输入）：
Tune-A-Video man-skiing源视频
mickey mouse is skiing on the snow：
spider man is skiing on the beach, cartoon style：
wonder woman, wearing a cowboy hat, is skiing：
a man, wearing pink clothes, is skiing at sunset：
iron man is skiing on the sea：
2. man-spraying-fire
A man is spraying fire（输入）：
TAV man-spraying-fire2源视频
A man is spraying water：
- RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by making sure all forward function outputs participate in calculating loss #78 ↩︎