MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMs

TL;DR

This paper introduces MI-Pruner, a novel crossmodal mutual information-based token pruning method for multimodal large language models, improving efficiency without architectural changes.

Contribution

It proposes a direct mutual information measurement approach for visual token pruning, outperforming attention-based methods in efficiency and effectiveness.

Findings

Outperforms previous attention-based pruning methods

Requires no internal attention maps or architectural modifications

Demonstrates minimal latency increase

Abstract

For multimodal large language models (MLLMs), visual information is relatively sparse compared with text. As a result, research on visual pruning emerges for efficient inference. Current approaches typically measure token importance based on the attention scores in the visual encoder or in the LLM decoder, then select visual tokens with high attention scores while pruning others. In this paper, we pursue a different and more surgical approach. Instead of relying on mechanism-specific signals, we directly compute Mutual Information (MI) between visual and textual features themselves, prior to their interaction. This allows us to explicitly measure crossmodal dependency at the feature levels. Our MI-Pruner is simple, efficient and non-intrusive, requiring no access to internal attention maps or architectural modifications. Experimental results demonstrate that our approach outperforms previous attention-based pruning methods with minimal latency.