STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

TL;DR

STAR introduces a two-stage, attention-guided token reduction framework for large vision-language models, significantly reducing inference costs while preserving or improving task performance through a global, holistic pruning approach.

Contribution

It proposes a novel, training-free, stage-wise token pruning method that considers the entire model's information flow, outperforming existing single-stage approaches.

Findings

Achieves substantial inference speedup with minimal performance loss.

Effectively preserves task-critical information through holistic pruning.

Demonstrates improved or comparable results across multiple architectures and benchmarks.

Abstract

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.