Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?

TL;DR

This paper critically examines token pruning methods in multimodal large language models, questioning their effectiveness and evaluation protocols, and provides insights for designing better token pruning strategies.

Contribution

The paper analyzes existing token pruning approaches, identifies their limitations, and offers insights to guide future research in improving token pruning techniques.

Findings

Many existing approaches underperform compared to naive random selection

Attention-based scoring may not reliably identify redundant tokens

Current evaluation protocols may be biased or incomplete

Abstract

Multimodal large language models (MLLMs) have shown remarkable performance for cross-modal understanding and generation, yet still suffer from severe inference costs. Recently, abundant works have been proposed to solve this problem with token pruning, which identifies the redundant tokens in MLLMs and then prunes them to reduce the computation and KV storage costs, leading to significant acceleration without training. While these methods claim efficiency gains, critical questions about their fundamental design and evaluation remain unanswered: Why do many existing approaches underperform even compared to naive random token selection? Are attention-based scoring sufficient for reliably identifying redundant tokens? Is language information really helpful during token pruning? What makes a good trade-off between token importance and duplication? Are current evaluation protocols comprehensive and unbiased? The ignorance of previous research on these problems hinders the long-term development of token pruning. In this paper, we answer these questions one by one, providing insights into the design of future token pruning methods.