IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs

TL;DR

This paper introduces IDPruner, a novel token pruning method for multimodal large language models that balances importance and diversity, leading to significant computational savings while maintaining high performance.

Contribution

IDPruner employs Maximal Marginal Relevance to optimally combine importance and diversity in visual token pruning without attention maps, enabling efficient and effective inference.

Findings

Retains 95.18% performance with 75% token pruning.

Maintains 86.40% performance under 90% pruning.

Achieves state-of-the-art results across multiple benchmarks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities, yet they encounter significant computational bottlenecks due to the massive volume of visual tokens. Consequently, visual token pruning, which substantially reduces the token count, has emerged as a critical technique for accelerating MLLM inference. Existing approaches focus on token importance, diversity, or an intuitive combination of both, without a principled framework for their optimal integration. To address this issue, we first conduct a systematic analysis to characterize the trade-off between token importance and semantic diversity. Guided by this analysis, we propose the \textbf{I}mportance and \textbf{D}iversity Pruner (\textbf{IDPruner}), which leverages the Maximal Marginal Relevance (MMR) algorithm to achieve a Pareto-optimal balance between these two objectives. Crucially, our method operates without requiring attention maps, ensuring full compatibility with FlashAttention and efficient deployment via one-shot pruning. We conduct extensive experiments across various model architectures and multimodal benchmarks, demonstrating that IDPruner achieves state-of-the-art performance and superior generalization across diverse architectures and tasks. Notably, on Qwen2.5-VL-7B-Instruct, IDPruner retains 95.18\% of baseline performance when pruning 75\% of the tokens, and still maintains 86.40\% even under an extreme 90\% pruning ratio. Our code is available at https://github.com/Tencent/AngelSlim.