GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning

TL;DR

GazeVLM introduces a novel multimodal architecture that internally controls attention to improve spatial reasoning and reduce hallucinations, achieving state-of-the-art performance in high-resolution multimodal tasks.

Contribution

It proposes GazeVLM, a model that internalizes active vision principles with autonomous gaze control, enhancing reasoning capabilities without external cropping or expanded context.

Findings

GazeVLM surpasses state-of-the-art VLMs by nearly 4% in its class.

It outperforms agentic multimodal pipelines by over 5% on HRBench datasets.

The model demonstrates improved spatial reasoning and reduced hallucinations.

Abstract

Human visual reasoning is governed by active vision, a process where metacognitive control drives top-down goal-directed attention, dynamically routing foveal focus toward task-relevant details while maintaining peripheral awareness of the global scene. In contrast, modern Vision-Language Models (VLMs) process visual information passively, relying on the static accumulation of massive token contexts that dilute spatial reasoning and induce linguistic hallucinations. Here we propose the following paradigm shift: GazeVLM, a multimodal architecture that internalizes this metacognitive oversight over its deployment of attention resources directly into the reasoning loop. By empowering the VLM to autonomously generate gaze tokens ($\texttt{<LOOK>}$), GazeVLM establishes a top-down control mechanism over its own causal attention mask. The model dynamically dictates its focal intent, triggering a continuous suppression bias that dampens irrelevant visual features, implementing spatial selective attention and simulating foveal fixation. Once local reasoning concludes, the bias lifts, seamlessly restoring the global view. This architecture enables the model to fluidly transition between global spatial awareness and localized focal reasoning without relying on external agentic contraptions like cropping tools, or inflating the context window with additional visual tokens derived from localized visual patches. Trained with a bespoke Group Relative Policy Optimization (GRPO) procedure that rewards valid grounding, our 4B-parameter GazeVLM delivers strong high-resolution multimodal reasoning performance, surpassing state-of-the-art VLMs in its parameter class by nearly 4% and agentic multimodal pipelines built around thinking with images by more than 5% on HRBench-4k and HRBench-8k.