Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding

TL;DR

Chain-of-Glimpse introduces a search-guided, multi-step reasoning framework for video understanding that explicitly grounds reasoning in visual evidence, improving interpretability and robustness.

Contribution

It proposes a novel, reinforcement learning-based, step-by-step reasoning method that explicitly anchors reasoning to visual evidence regions in videos.

Findings

Achieves consistent performance improvements on multiple video reasoning benchmarks.

Demonstrates robustness and generalization across diverse datasets.

Provides interpretable multi-step reasoning trajectories.

Abstract

Video understanding requires identifying and reasoning over semantically discriminative visual objects across frames, yet existing object-agnostic solutions struggle to effectively handle substantial object variations over time. To address this, we introduce Chain-of-Glimpse, a search-guided progressive object-grounded reasoning framework that explicitly anchors each reasoning step to specific visual evidence regions, enabling compositional and multi-step decision-making. Formally, Chain-of-Glimpse formulates video reasoning as a step-by-step process that incrementally builds spatially grounded traces around task-relevant visual objects, thereby mitigating over-reliance on saliency-driven cues. Specifically, Chain-of-Glimpse features a search-guided controller, optimized via reinforcement learning with a format reward that significantly incentivizes grounding capability, to iteratively ground visual evidence regions and form reliable reasoning trajectories, yielding accurate and interpretable multi-step decisions. Extensive evaluations on both in domain NExTQA and out-of-domain Video-Holmes, CG-Bench Reasoning, and VRBench benchmarks demonstrate consistent performance gains, robustness and generalization of Chain-of-Glimpse across diverse video reasoning tasks.