Long-CODE: Isolating Pure Long-Context as an Orthogonal Dimension in Video Evaluation

TL;DR

This paper introduces Long-CODE, a new framework and benchmark for evaluating long-video generation quality, focusing on long-range attributes like narrative consistency, which existing short-video metrics overlook.

Contribution

The paper proposes a novel long-video evaluation metric based on shot dynamics and introduces a dedicated dataset with human annotations for long-range video assessment.

Findings

The new metric correlates highly with human judgments.

Existing short-video metrics are insensitive to structural long-range inconsistencies.

Long-CODE provides a comprehensive benchmark for long-video evaluation.

Abstract

As video generation models achieve unprecedented capabilities, the demand for robust video evaluation metrics becomes increasingly critical. Traditional metrics are intrinsically tailored for short-video evaluation, predominantly assessing frame-level visual quality and localized temporal smoothness. However, as state-of-the-art video generation models scale to generate longer videos, these metrics fail to capture essential long-range characteristics, such as narrative richness and global causal consistency. Recognizing that short-term visual perception and long-context attributes are fundamentally orthogonal dimensions, we argue that long-video metrics should be disentangled from short-video assessments. In this paper, we focus on the rigorous justification and design of a dedicated framework for long-video evaluation. We first introduce a suite of long-video attribute corruption tests, exposing the critical limitations of existing hort-video metrics from their insensitivity to structural inconsistencies, such as shot-level perturbations and narrative shuffling. To bridge this gap, we design a novel long-video metric based on shot dynamics, which is highly sensitive to the long-range testing framework. Furthermore, we introduce Long-CODE (Long-Context as an Orthogonal Dimension for video Evaluation), a specialized dataset designed to benchmark long-video evaluation, with human annotations isolated specifically to genuine long-range characteristics. Extensive experiments show that our proposed metrics achieve state-of-the-art correlation with human judgments. Ultimately, our metric and benchmark seamlessly complement existing short-video standards, establishing a holistic and unbiased evaluation paradigm for video generation models.