Spotlight: Identifying and Localizing Video Generation Errors Using VLMs

TL;DR

This paper introduces Spotlight, a new task for localizing and explaining errors in text-to-video generation, revealing current models' limitations and proposing strategies to improve error detection.

Contribution

The paper presents a novel task and dataset for error localization in video generation, along with evaluation of current models and strategies to enhance their performance.

Findings

Current VLMs lag behind humans in error detection.

Adherence and physics errors are most common and persistent.

Inference strategies can nearly double VLM performance.

Abstract

Current text-to-video models (T2V) can generate high-quality, temporally coherent, and visually realistic videos. Nonetheless, errors still often occur, and are more nuanced and local compared to the previous generation of T2V models. While current evaluation paradigms assess video models across diverse dimensions, they typically evaluate videos holistically without identifying when specific errors occur or describing their nature. We address this gap by introducing Spotlight, a novel task aimed at localizing and explaining video-generation errors. We generate 600 videos using 200 diverse textual prompts and three state-of-the-art video generators (Veo 3, Seedance, and LTX-2), and annotate over 1600 fine-grained errors across six types, including motion, physics, and prompt adherence. We observe that adherence and physics errors are predominant and persist across longer segments, whereas appearance-disappearance and body pose errors manifest in shorter segments. We then evaluate current VLMs on Spotlight and find that VLMs lag significantly behind humans in error identification and localization in videos. We propose inference-time strategies to probe the limits of current VLMs on our task, improving performance by nearly 2x. Our task paves a way forward to building fine-grained evaluation tools and more sophisticated reward models for video generators.