Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?

TL;DR

This paper critically examines video understanding benchmarks, revealing limitations like language priors and shuffling invariance, and introduces VBenchComp, a pipeline for more precise evaluation of models' temporal reasoning abilities.

Contribution

The paper identifies key issues in current benchmarks and proposes VBenchComp, a novel pipeline that categorizes questions to better assess models' true temporal understanding.

Findings

Models often rely on language priors, not video content.

Shuffling frames does not always degrade model performance.

VBenchComp enables fine-grained evaluation of temporal reasoning.

Abstract

Existing video understanding benchmarks often conflate knowledge-based and purely image-based questions, rather than clearly isolating a model's temporal reasoning ability, which is the key aspect that distinguishes video understanding from other modalities. We identify two major limitations that obscure whether higher scores truly indicate stronger understanding of the dynamic content in videos: (1) strong language priors, where models can answer questions without watching the video; and (2) shuffling invariance, where models maintain similar performance on certain questions even when video frames are temporally shuffled. To alleviate these issues, we propose VBenchComp, an automated pipeline that categorizes questions into different domains: LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions can be answered without viewing the video; Semantic questions remain answerable even when the video frames are shuffled; and Temporal questions require understanding the correct temporal order of frames. The rest of the questions are labeled as Others. This can enable fine-grained evaluation of different capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that are hidden by traditional overall scores, and we offer insights and recommendations for designing future benchmarks that more accurately assess video LLMs.