A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs

TL;DR

The paper introduces MVP, a new shortcut-aware video QA benchmark with minimal video pairs to accurately evaluate physical understanding in video language models, reducing bias from superficial cues.

Contribution

It presents the MVP benchmark with minimal-change pairs to mitigate shortcut solutions, enabling more reliable assessment of physical reasoning in video models.

Findings

Human performance is 92.9% on MVP.

State-of-the-art models achieve only 40.2%.

Models struggle with physical reasoning beyond superficial cues.

Abstract

Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.