VideoGPT:使用VQ-VAE和Transformers的视频生成

1 Title

VideoGPT: Video Generation using VQ-VAE and Transformers(Wilson Yan,Yunzhi Zhang ,Pieter Abbeel,Aravind Srinivas)

2 Conlusion

This paper present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings.

3 Good Sentences

1、High-fidelity natural videos is one notable modality that has not seen the same level of progress in generative modeling as compared to images, audio, and text. This is reasonable since the complexity of natural videos requires modeling correlations across both space and time with much higher input dimensions. Video modeling is therefore a natural next challenge for current deep generative models. (The significance of this work)

2、The above line of reasoning leads us to our proposed model:VideoGPT, a simple video generation architecture that is a minimal adaptation of VQ-VAE and GPT architectures for videos.(The reason for choosing VideoGPT)

3、Although the VQ-VAE is trained unconditionally, we can generate conditional samples by training a conditional prior. We use two types of conditioning:Cross Attention and Conditional Norms.(How to transform unconditional to conditional learning)


背景知识

VQ-VAE

VQ-VAE能利用codebook机制把图像编码成离散向量

Method

整个训练过程如图所示,分为两个部分,训练VQ-VAE(左)和训练隐空间中的自回归Transformer(右)

第一阶段与原始VQ-VAE训练过程类似。

第二阶段,VQ-VAE将视频数据编码为隐序列作为先验模型的训练数据。首先从先验中采样隐序列,然后使用VQ-VAE将隐序列解码为视频样本。(Transformer的作用是引入条件,这里可以使用交叉注意力或者Conditional Norms:)

相关推荐
阿庆_AI研发工程师6 小时前
Stanford CS336 Lecture 10:大语言模型推理与性能优化
算法
用户7341681035486 小时前
基于MindSpore的深度残差收缩网络程序
算法
Pniubi7 小时前
力扣239.滑动窗口最大值(优先队列)
算法·leetcode·职场和发展
青少儿编程课堂8 小时前
第十八届蓝桥杯青少组图形化真题 + 全国青少年信息素养大赛 + 电子学会图形化考级:校园泡泡堂大挑战解析
c++·python·算法·bfs·信息学竞赛
超低空9 小时前
一文说清LRU算法
后端·算法·架构
青少儿编程课堂9 小时前
蓝桥杯·全国青少年信息素养大赛·电子学会Python/C++考级:Dinic最大流解析
c++·python·算法·bfs·信息学竞赛
开开心心就好10 小时前
显示器关不掉怎么办,用小工具一键息屏
算法·智能手机·pdf·计算机外设·ocr·lua·1024程序员节
专注仿真10 小时前
基于 AFSIM 的智能体训练:多智能体强化学习篇 —— QTRAN 算法实战
开发语言·python·算法·php·强化学习·afsim智能体·仿真训练
FR02EN77710 小时前
[LeetCode] 26、80.删除有序数组中的重复项 题解
数据结构·算法
forest_091 天前
《AI 编程 Harness 实战手册》04|审批这道门:点了一百次“允许”,还剩多少作用?
人工智能·算法·prompt·agi