Video-CoM: Interactive Video Reasoning via Chain of Manipulations

TL;DR

Video-CoM introduces an interactive video reasoning paradigm where models actively manipulate video content through iterative actions, enabling deeper understanding and reasoning, and achieves state-of-the-art results with fewer training samples.

Contribution

The paper proposes a novel interactive reasoning framework with a chain of manipulations, supported by a new instruction dataset and reinforcement learning, advancing video understanding beyond passive analysis.

Findings

Achieves 3.6% performance improvement over state-of-the-art models.

Uses fewer training samples than comparable models.

Reasoning-aware rewards enhance accuracy and interpretability.

Abstract

Recent multimodal large language models (MLLMs) have advanced video understanding, yet most still "think about videos" ie once a video is encoded, reasoning unfolds entirely in text, treating visual input as a static context. This passive paradigm creates a semantic bottleneck: models cannot rewatch, refocus, or verify evidence, leading to shallow visual reasoning on tasks requiring fine grained spatio temporal understanding. In this work, we introduce Interactive Video Reasoning, a new paradigm that transforms video into an active cognitive workspace, enabling models to "think with videos". Our model, Video CoM, reasons through a Chain of Manipulations (CoM), performing iterative visual actions to gather and refine evidence. To support this behavior, we construct Video CoM Instruct, an 18K instruction tuning dataset curated for multi step manipulation reasoning. Beyond supervised learning, we further optimize the manipulation policy via reinforcement learning with reasoning aware Group Relative Policy Optimization (GRPO). Unlike prior work that relies solely on sparse answer rewards, our method introduces step level reasoning rewards, guiding the model toward grounded and consistent reasoning. Video CoM achieves strong results across nine video reasoning benchmarks, improving average performance by 3.6 percent over recent state of the art models, while training on only 25K SFT and 3K GRPO video samples, significantly fewer than comparable large scale models. Ablation studies demonstrate that reasoning aware rewards improve both accuracy and interpretability. Code: https://github.com/mbzuai-oryx/Video-CoM