Learning to See What You Need: Gaze Attention for Multimodal Large Language Models

TL;DR

This paper introduces Gaze Attention, a mechanism for multimodal large language models that selectively attends to relevant visual regions, reducing computation and improving focus without losing global context.

Contribution

It proposes a novel gaze-based attention method that dynamically focuses on task-relevant visual regions, significantly reducing computational overhead while maintaining or improving performance.

Findings

Gaze Attention matches or surpasses dense-attention baselines.

Reduces attention computation by up to 90%.

Maintains holistic visual awareness with learnable context tokens.

Abstract

When humans describe a visual scene, they do not process the entire image uniformly; instead, they selectively fixate on regions relevant to their intended description. In contrast, current multimodal large language models (MLLMs) attend to all visual tokens at each generation step, leading to diluted focus and unnecessary computational overhead. In this work, we introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation. Specifically, we spatially group visual embeddings-stored as key-value caches-into compact gaze regions, each represented by a lightweight descriptor. At each decoding step, the model dynamically selects the most relevant regions and restricts attention to them, reducing redundant computation while enhancing focus. To mitigate the loss of global context caused by localized attention, we further propose learnable context tokens appended to each image or frame, allowing the model to maintain holistic visual awareness. Extensive experiments on image and video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.