IPCV: Information-Preserving Compression for MLLM Visual Encoders

TL;DR

IPCV is a training-free, information-preserving token compression framework for multimodal large language model visual encoders that reduces computation while maintaining performance by selectively pruning tokens with minimal information loss.

Contribution

IPCV introduces a novel, training-free token pruning method with neighbor-guided reconstruction and attention stabilization to improve efficiency of MLLM visual encoders.

Findings

Significantly reduces computational cost in MLLM visual encoders.

Outperforms existing training-free token compression methods.

Effective across diverse image and video benchmarks.

Abstract

Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruning strategies are inadequate: LLM-stage token pruning overlooks the ViT's overhead, while conventional ViT token pruning, without language guidance, risks discarding textually critical visual cues and introduces feature distortions amplified by the ViT's bidirectional attention. To meet these challenges, we propose IPCV, a training-free, information-preserving compression framework for MLLM visual encoders. IPCV enables aggressive token pruning inside the ViT via Neighbor-Guided Reconstruction (NGR) that temporarily reconstructs pruned tokens to participate in attention with minimal overhead, then fully restores them before passing to the LLM. Besides, we introduce Attention Stabilization (AS) to further alleviate the negative influence from token pruning by approximating the K/V of pruned tokens. It can be directly applied to previous LLM-side token pruning methods to enhance their performance. Extensive experiments show that IPCV substantially reduces end-to-end computation and outperforms state-of-the-art training-free token compression methods across diverse image and video benchmarks. Our code is available at https://github.com/Perkzi/IPCV.