Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning

TL;DR

This paper introduces Glance-or-Gaze, a reinforcement learning framework for large multimodal models that actively select visual regions to improve complex visual query answering.

Contribution

It proposes a novel Selective Gaze mechanism and a dual-stage training strategy to enhance visual search and reasoning in multimodal models.

Findings

Achieves state-of-the-art results across six benchmarks.

Both Selective Gaze and complexity-adaptive RL are crucial for performance.

Reduces visual redundancy and noise in search processes.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in visual understanding, yet they struggle with knowledge-intensive queries involving long-tail entities or evolving information due to static parametric knowledge. Recent search-augmented approaches attempt to address this limitation, but existing methods rely on indiscriminate whole-image retrieval that introduces substantial visual redundancy and noise, and lack deep iterative reflection, limiting their effectiveness on complex visual queries. To overcome these challenges, we propose Glance-or-Gaze (GoG), a fully autonomous framework that shifts from passive perception to active visual planning. GoG introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions, filtering irrelevant information before retrieval. We design a dual-stage training strategy: Reflective GoG Behavior Alignment via supervised fine-tuning instills the fundamental GoG paradigm, while Complexity-Adaptive Reinforcement Learning further enhances the model's capability to handle complex queries through iterative reasoning. Experiments across six benchmarks demonstrate state-of-the-art performance. Ablation studies confirm that both Selective Gaze and complexity-adaptive RL are essential for effective visual search.