SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generation

TL;DR

SLVMEval is a benchmark for evaluating text-to-long-video systems by synthetic degradation and crowdsourced pairwise comparisons, revealing current evaluation weaknesses compared to human judgment.

Contribution

The paper introduces a synthetic, controlled benchmark for meta-evaluating long-video assessment systems, highlighting their shortcomings relative to human evaluators.

Findings

Humans achieve 84.7%-96.8% accuracy in video quality assessment.

Existing evaluation systems underperform humans in 9 out of 10 aspects.

SLVMEval provides a controlled testbed for assessing evaluation system reliability.

Abstract

This paper proposes the synthetic long-video meta-evaluation (SLVMEval), a benchmark for meta-evaluating text-to-video (T2V) evaluation systems. The proposed SLVMEval benchmark focuses on assessing these systems on videos of up to 10,486 s (approximately 3 h). The benchmark targets a fundamental requirement, namely, whether the systems can accurately assess video quality in settings that are easy for humans to assess. We adopt a pairwise comparison-based meta-evaluation framework. Building on dense video-captioning datasets, we synthetically degrade source videos to create controlled "high-quality versus low-quality" pairs across 10 distinct aspects. Then, we employ crowdsourcing to filter and retain only those pairs in which the degradation is clearly perceptible, thereby establishing an effective final testbed. Using this testbed, we assess the reliability of existing evaluation systems in ranking these pairs. Experimental results demonstrate that human evaluators can identify the better long video with 84.7%-96.8% accuracy, and in nine of the 10 aspects, the accuracy of these systems falls short of human assessment, revealing weaknesses in text-to-long-video evaluation.