Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models

TL;DR

This paper introduces REVEAL{}, a benchmark that uncovers significant weaknesses in current Video-Language Models, revealing their poor understanding of video content, temporal dynamics, and motion through stress tests.

Contribution

The paper presents a new diagnostic benchmark with automated data generation to evaluate and expose fundamental weaknesses in contemporary VidLMs.

Findings

Models often misinterpret reversed scenes as forward

Models answer questions ignoring video content

Models struggle with camera motion and spatiotemporal occlusion

Abstract

This work investigates a fundamental question: Do Video-Language Models (VidLMs) robustly account for video content, temporal sequence, and motion? Our investigation shows that, surprisingly, they often do not. We introduce REVEAL{}, a diagnostic benchmark that probes fundamental weaknesses of contemporary VidLMs through five controlled stress tests; assessing temporal expectation bias, reliance on language-only shortcuts, video sycophancy, camera motion sensitivity, and robustness to spatiotemporal occlusion. We test leading open- and closed-source VidLMs and find that these models confidently describe reversed scenes as forward, answer questions while neglecting video content, agree with false claims, struggle with basic camera motion, and fail to aggregate temporal information amidst simple spatiotemporal masking. Humans, on the other hand, succeed at these tasks with ease. Alongside our benchmark, we provide a data pipeline that automatically generates diagnostic examples for our stress tests, enabling broader and more scalable evaluation. We will release our benchmark and code to support future research.