NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation

TL;DR

NarrLV introduces a comprehensive benchmark for evaluating the narrative expression capabilities of long video generation models, inspired by film narrative theory, using novel metrics and prompt generation methods.

Contribution

This paper presents the first benchmark specifically designed to evaluate narrative content in long video generation, incorporating novel metrics and an automatic prompt generation pipeline.

Findings

The proposed metric aligns closely with human judgments.

Current models have limited capabilities in narrative content expression.

NarrLV reveals detailed capability boundaries of existing models.

Abstract

With the rapid development of foundation video generation technologies, long video generation models have exhibited promising research potential thanks to expanded content creation space. Recent studies reveal that the goal of long video generation tasks is not only to extend video duration but also to accurately express richer narrative content within longer videos. However, due to the lack of evaluation benchmarks specifically designed for long video generation models, the current assessment of these models primarily relies on benchmarks with simple narrative prompts (e.g., VBench). To the best of our knowledge, our proposed NarrLV is the first benchmark to comprehensively evaluate the Narrative expression capabilities of Long Video generation models. Inspired by film narrative theory, (i) we first introduce the basic narrative unit maintaining continuous visual presentation in videos as Temporal Narrative Atom (TNA), and use its count to quantitatively measure narrative richness. Guided by three key film narrative elements influencing TNA changes, we construct an automatic prompt generation pipeline capable of producing evaluation prompts with a flexibly expandable number of TNAs. (ii) Then, based on the three progressive levels of narrative content expression, we design an effective evaluation metric using the MLLM-based question generation and answering framework. (iii) Finally, we conduct extensive evaluations on existing long video generation models and the foundation generation models. Experimental results demonstrate that our metric aligns closely with human judgments. The derived evaluation outcomes reveal the detailed capability boundaries of current video generation models in narrative content expression.