视频生成【文章汇总】SVD, Sora, Latte, VideoCrafter12, DiT...

视频生成【文章汇总】SVD, Sora, Latte, VideoCrafter12, DiT...

  • [【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](#【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions)
  • [【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models](#【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models)
  • [【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](#【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation)
  • [【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation](#【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)

数据集

指标

【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions

Authors: Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, Ying Shan
Abstract Sora's high-motion intensity and long consistent videos have significantly impacted the field of video generation, attracting unprecedented attention. However, existing publicly available datasets are inadequate for generating Sora-like videos, as they mainly contain short videos with low motion intensity and brief captions. To address these issues, we propose MiraData, a high-quality video dataset that surpasses previous ones in video duration, caption detail, motion strength, and visual quality. We curate MiraData from diverse, manually selected sources and meticulously process the data to obtain semantically consistent clips. GPT-4V is employed to annotate structured captions, providing detailed descriptions from four different perspectives along with a summarized dense caption. To better assess temporal consistency and motion intensity in video generation, we introduce MiraBench, which enhances existing benchmarks by adding 3D consistency and tracking-based motion strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics covering temporal consistency, motion strength, 3D consistency, visual quality, text-video alignment, and distribution similarity. To demonstrate the utility and effectiveness of MiraData, we conduct experiments using our DiT-based video generation model, MiraDiT. The experimental results on MiraBench demonstrate the superiority of MiraData, especially in motion strength. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】



【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models

Authors: Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu
Abstract Video generation has witnessed significant advancements, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal evaluation system should provide insights to inform future developments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has three appealing properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity inconsistency, motion smoothness, temporal flickering, and spatial relationship, etc). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investigate the gaps between video and image generation models. We will open-source VBench, including all prompts, evaluation methods, generated videos, and human preference annotations, and also include more video generation models in VBench to drive forward the field of video generation. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读】

【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation

Authors: Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, Xihui Liu
Abstract Text-to-video (T2V) generation models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of MLLM-based metrics, detection-based metrics, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 700 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and different compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope that our attempt will shed light on future research in this direction. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读】


【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation

Authors: Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, Yu Qiao
Abstract We propose a novel Latent Diffusion Transformer, namely Latte, for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to text-to-video generation (T2V) task, where Latte achieves comparable results compared to recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

相关推荐
9527华安2 小时前
FPGA实现PCIE3.0视频采集转10G万兆UDP网络输出,基于XDMA+GTH架构,提供工程源码和技术支持
网络·fpga开发·udp·音视频·xdma·pcie3.0·万兆网
电子科技圈2 小时前
XMOS携手合作伙伴晓龙国际联合推出集成了ASRC等功能的多通道音频板
科技·嵌入式硬件·mcu·物联网·音视频·iot
码码哈哈0.02 小时前
免费的视频混剪综合处理工具介绍与下载
音视频
莫固执,朋友3 小时前
网络抓包工具tcpdump 在海思平台上的编译使用
网络·ffmpeg·音视频·tcpdump
深海呐3 小时前
Android 从本地选择视频,用APP播放或进行其他处理
android·音视频·从本地选择视频,用app播放·从本地选择视频,并拿到信息·跳转到本地视频列表
cuijiecheng20183 小时前
音视频入门基础:MPEG2-TS专题(6)——FFmpeg源码中,获取MPEG2-TS传输流每个transport packet长度的实现
ffmpeg·音视频
安静读书6 小时前
Java解析视频FPS(帧率)、分辨率信息
java·python·音视频
VisionX Lab8 小时前
数据脱敏工具:基于 FFmpeg 的视频批量裁剪
python·ffmpeg·音视频
EasyNTS14 小时前
RTSP播放器EasyPlayer.js播放器分辨率高的视频在设置container的宽高较小时,会出现锯齿状的画面效果
音视频
算家云16 小时前
VideoCrafter模型部署教程
人工智能·深度学习·机器学习·显卡·算力·视频生成·ai视频编辑