视频生成【文章汇总】SVD, Sora, Latte, VideoCrafter12, DiT...

视频生成【文章汇总】SVD, Sora, Latte, VideoCrafter12, DiT...

  • [【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](#【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions)
  • [【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models](#【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models)
  • [【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](#【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation)
  • [【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation](#【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)
  • [【arxiv 2024】xxx](#【arxiv 2024】xxx)

数据集

指标

【arXiv 2024】MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions

Authors: Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, Ying Shan
Abstract Sora's high-motion intensity and long consistent videos have significantly impacted the field of video generation, attracting unprecedented attention. However, existing publicly available datasets are inadequate for generating Sora-like videos, as they mainly contain short videos with low motion intensity and brief captions. To address these issues, we propose MiraData, a high-quality video dataset that surpasses previous ones in video duration, caption detail, motion strength, and visual quality. We curate MiraData from diverse, manually selected sources and meticulously process the data to obtain semantically consistent clips. GPT-4V is employed to annotate structured captions, providing detailed descriptions from four different perspectives along with a summarized dense caption. To better assess temporal consistency and motion intensity in video generation, we introduce MiraBench, which enhances existing benchmarks by adding 3D consistency and tracking-based motion strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics covering temporal consistency, motion strength, 3D consistency, visual quality, text-video alignment, and distribution similarity. To demonstrate the utility and effectiveness of MiraData, we conduct experiments using our DiT-based video generation model, MiraDiT. The experimental results on MiraBench demonstrate the superiority of MiraData, especially in motion strength. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】



【CVPR 2024】VBench : Comprehensive Benchmark Suite for Video Generative Models

Authors: Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu
Abstract Video generation has witnessed significant advancements, yet evaluating these models remains a challenge. A comprehensive evaluation benchmark for video generation is indispensable for two reasons: 1) Existing metrics do not fully align with human perceptions; 2) An ideal evaluation system should provide insights to inform future developments of video generation. To this end, we present VBench, a comprehensive benchmark suite that dissects "video generation quality" into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. VBench has three appealing properties: 1) Comprehensive Dimensions: VBench comprises 16 dimensions in video generation (e.g., subject identity inconsistency, motion smoothness, temporal flickering, and spatial relationship, etc). The evaluation metrics with fine-grained levels reveal individual models' strengths and weaknesses. 2) Human Alignment: We also provide a dataset of human preference annotations to validate our benchmarks' alignment with human perception, for each evaluation dimension respectively. 3) Valuable Insights: We look into current models' ability across various evaluation dimensions, and various content types. We also investigate the gaps between video and image generation models. We will open-source VBench, including all prompts, evaluation methods, generated videos, and human preference annotations, and also include more video generation models in VBench to drive forward the field of video generation. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读】

【arxiv 2024】T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation

Authors: Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, Xihui Liu
Abstract Text-to-video (T2V) generation models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of MLLM-based metrics, detection-based metrics, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 700 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and different compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope that our attempt will shed light on future research in this direction. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读】


【arxiv 2024】Latte: Latent Diffusion Transformer for Video Generation

Authors: Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, Yu Qiao
Abstract We propose a novel Latent Diffusion Transformer, namely Latte, for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to text-to-video generation (T2V) task, where Latte achieves comparable results compared to recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation. 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

【arxiv 2024】xxx

Authors:
Abstract 【Paper】 > 【Github_Code】 > 【Project】 > 【中文解读,待续】

相关推荐
山顶夕景1 天前
【ECCV2026】流式视频理解Thinking in Streaming Video
音视频·vlm·视频理解·多模态理解·流式视频
zhanlang02172 天前
HTML 零基础入门到企业实战【第五章 5.4】内嵌网页 iframe 标签:嵌入网页、视频、地图实战
前端·html·音视频
TengTaiTech2 天前
头戴耳机做运动相机外接麦:一颗麦克风同时服务蓝牙通话与 3.5mm 录音的底层重构
数码相机·音视频·蓝牙·qcc
2301_790355973 天前
【技术拆解】短视频配音自动化流程:从文案预处理到 SRT 字幕的 9 个环节
自然语言处理·音视频
大飞哥软件自习室3 天前
视频批量截图并自动拼图是怎么做的:从四步向导、三种截帧模式到等距抽样与日志兜底的实现拆解,附 FFmpeg 调用与线程模型说明
ffmpeg·音视频·批量截图
Black蜡笔小新4 天前
消防控制室/配电房/中控室...视频汇聚平台EasyCVR岗位AI智能监测方案守住关键岗位
大数据·音视频·easycvr
FlightYe5 天前
音视频修炼之容器与协议(四):RTSP详解
android·linux·ffmpeg·音视频·dash
乐橙开放平台5 天前
海康和大华摄像头能统一管理吗?可以通过 GB28181 或视频融合平台接入
网络·后端·物联网·音视频
Mirawork5 天前
Mirawork v0.32.0 更新体验:会话引用、任务分享与音频转写怎么用
人工智能·音视频
熬夜磕代码丶6 天前
RN鸿蒙空间音频实战
华为·音视频·harmonyos