D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning

TL;DR

D2Pruner is a novel token pruning framework for Multimodal Large Language Models that combines debiased importance and structural diversity to significantly reduce computational costs while maintaining high performance, especially on localization tasks.

Contribution

It introduces a unique combination of debiased importance scoring and a Maximal Independent Set-based structural pruning mechanism for improved token selection.

Findings

Reduces FLOPs by 74.2% with 99.2% performance retention on general tasks.

Maintains 85.7% performance at 90% token reduction on localization benchmarks.

Outperforms existing methods with up to 63.53% improvement.

Abstract

Processing long visual token sequences poses a significant computational burden on Multimodal Large Language Models (MLLMs). While token pruning offers a path to acceleration, we find that current methods, while adequate for general understanding, catastrophically fail on fine-grained localization tasks. We attribute this failure to the inherent flaws of the two prevailing strategies: importance-based methods suffer from a strong positional bias, an inherent model artifact that distracts from semantic content, while diversity-based methods exhibit structural blindness, disregarding the user's prompt and spatial redundancy. To address this, we introduce D2Pruner, a framework that rectifies these issues by uniquely combining debiased importance with a structural pruning mechanism. Our method first secures a core set of the most critical tokens as pivots based on a debiased attention score. It then performs a Maximal Independent Set (MIS) selection on the remaining tokens, which are modeled on a hybrid graph where edges signify spatial proximity and semantic similarity. This process iteratively preserves the most important and available token while removing its neighbors, ensuring that the supplementary tokens are chosen to maximize importance and diversity. Extensive experiments demonstrate that D2Pruner has exceptional efficiency and fidelity. Applied to LLaVA-1.5-7B for general understanding tasks, it reduces FLOPs by 74.2\% while retaining 99.2\% of its original performance. Furthermore, in challenging localization benchmarks with InternVL-2.5-8B, it maintains 85.7\% performance at a 90\% token reduction rate, marking a significant advancement with up to 63. 53\% improvement over existing methods.