MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

TL;DR

MMR-V introduces a challenging benchmark for evaluating multimodal deep reasoning in videos, emphasizing long-range, multi-frame inference and reasoning beyond perception, revealing current models' limitations.

Contribution

This paper presents MMR-V, a novel benchmark designed to evaluate deep multimodal reasoning in videos, addressing gaps in existing datasets by focusing on long-range, hidden, and confusable reasoning tasks.

Findings

Current models achieve only around 52.5% accuracy on MMR-V.

Reasoning strategies like Chain-of-Thought have limited impact.

Multi-modal reasoning differs from textual reasoning, affecting performance.

Abstract

The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, o4-mini, achieves only 52.5% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Further analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.