Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models

TL;DR

This paper introduces SPpruner, a subject-centric progressive visual token reduction method for vision-language models that improves efficiency while maintaining accuracy by mimicking human visual focus and context understanding.

Contribution

The paper proposes a novel focus-then-context paradigm with explicit saliency modeling and contextual aggregation, enhancing token reduction effectiveness in vision-language models.

Findings

Achieves up to 2.53x speedup with only 22.2% tokens retained in Qwen2.5-VL.

Reduces FLOPs by 67% on LLaVA with minimal accuracy loss.

Outperforms state-of-the-art token reduction methods across multiple benchmarks.

Abstract

Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference. Existing vision token reduction methods alleviate this burden, but they unintentionally preserve the isolated visual subject strictly aligned with the user's query, which fails to substantially explore salient subjects and their contextual relationships. In this paper, we propose SPpruner, a subject-centric progressive reduction paradigm that emulates the \textit{Focus-then-Context} mechanism of the human visual perception system. Specifically, we first construct a focus identification module to explicitly model the interplay between visual saliency and semantic relevance. Herein, it can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input. Subsequently, a context-aware structural scanning module is developed to aggregate contextual cues from neighboring regions. As such, it can effectively restore global relational dependencies to uphold the structural integrity of the preserved subjects. Extensive experiments demonstrate that our paradigm consistently outperforms SOTA methods, achieving up to 2.53 times speedup with only 22.2% of visual tokens retained in Qwen2.5-VL and a 67% FLOPs reduction on LLaVA with a negligible 0.6% accuracy drop.