TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

TL;DR

TimeBlind is a diagnostic benchmark designed to evaluate and improve the spatio-temporal understanding capabilities of multimodal large language models in video reasoning tasks, revealing current models' reliance on static cues.

Contribution

We introduce TimeBlind, a novel benchmark that isolates temporal reasoning from static recognition, providing a clear assessment of models' true spatio-temporal understanding.

Findings

State-of-the-art models achieve only 48.2% accuracy on TimeBlind.

Models rely heavily on static visual shortcuts rather than temporal reasoning.

TimeBlind exposes significant gaps in current video understanding models.

Abstract

Fine-grained spatio-temporal understanding is essential for video reasoning and embodied AI. Yet, while Multimodal Large Language Models (MLLMs) master static semantics, their grasp of temporal dynamics remains brittle. We present TimeBlind, a diagnostic benchmark for compositional spatio-temporal understanding. Inspired by cognitive science, TimeBlind categorizes fine-grained temporal understanding into three levels: recognizing atomic events, characterizing event properties, and reasoning about event interdependencies. Unlike benchmarks that conflate recognition with temporal reasoning, TimeBlind leverages a minimal-pairs paradigm: video pairs share identical static visual content but differ solely in temporal structure, utilizing complementary questions to neutralize language priors. Evaluating over 20 state-of-the-art MLLMs (e.g., GPT-5, Gemini 3 Pro) on 600 curated instances (2400 video-question pairs), reveals that the Instance Accuracy (correctly distinguishing both videos in a pair) of the best performing MLLM is only 48.2%, far below the human performance (98.2%). These results demonstrate that even frontier models rely heavily on static visual shortcuts rather than genuine temporal logic, positioning TimeBlind as a vital diagnostic tool for next-generation video understanding. Dataset and code are available at https://baiqi-li.github.io/timeblind_project/ .