Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs

TL;DR

This paper introduces PromPrune, an adaptive visual token selection method for VLMs that balances saliency and coverage based on sample-specific semantic prominence, significantly reducing computation while maintaining accuracy.

Contribution

The paper proposes a novel sample-adaptive token compression framework that dynamically balances local saliency and global coverage in VLMs, outperforming static methods.

Findings

Reduces FLOPs by 88% on LLaVA-NeXT-7B

Decreases prefill latency by 22%

Maintains 97.5% of original accuracy

Abstract

Large Vision-Language Models (VLMs) achieve strong multimodal understanding capabilities by leveraging high-resolution visual inputs, but the resulting large number of visual tokens creates a major computational bottleneck. Recent work mitigates this issue through visual token compression, typically compressing tokens based on saliency, diversity, or a fixed combination of both. We observe that the distribution of semantic prominence varies substantially across samples, leading to different optimal trade-offs between local saliency preservation and global coverage. This observation suggests that applying a static compression strategy across all samples can be suboptimal. Motivated by this insight, we propose PromPrune, a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline. Our method adaptively balances local saliency preservation and global coverage according to the semantic prominence distribution of each sample. By allocating token budgets between locally salient regions and globally diverse regions, our method maintains strong performance even under high compression ratios. On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.