MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation

TL;DR

MovieBench is a new hierarchical dataset of long, coherent movies designed to facilitate research and development in long video generation models, addressing current limitations in scene complexity and character consistency.

Contribution

The paper introduces MovieBench, a comprehensive dataset with hierarchical structure, multi-scene narratives, and character consistency, specifically tailored for long video generation research.

Findings

MovieBench enables analysis of character consistency across scenes.

The dataset reveals challenges in maintaining narrative coherence.

Experiments show improved understanding of long video generation complexities.

Abstract

Recent advancements in video generation models, like Stable Video Diffusion, show promising results, but primarily focus on short, single-scene videos. These models struggle with generating long videos that involve multiple scenes, coherent narratives, and consistent characters. Furthermore, there is no publicly available dataset tailored for the analysis, evaluation, and training of long video generation models. In this paper, we present MovieBench: A Hierarchical Movie-Level Dataset for Long Video Generation, which addresses these challenges by providing unique contributions: (1) movie-length videos featuring rich, coherent storylines and multi-scene narratives, (2) consistency of character appearance and audio across scenes, and (3) hierarchical data structure contains high-level movie information and detailed shot-level descriptions. Experiments demonstrate that MovieBench brings some new insights and challenges, such as maintaining character ID consistency across multiple scenes for various characters. The dataset will be public and continuously maintained, aiming to advance the field of long video generation. Data can be found at: https://weijiawu.github.io/MovieBench/.