TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model

TL;DR

TransPrune introduces a novel, training-free token pruning method for large vision-language models that leverages token transition signals to efficiently reduce computational costs while maintaining high performance.

Contribution

It proposes a new token importance criterion based on token transitions, overcoming limitations of attention-based methods, and demonstrates its effectiveness across multiple benchmarks.

Findings

TransPrune reduces inference TFLOPs by over 50%.

It achieves comparable performance to original LVLMs on eight benchmarks.

TTV alone is an effective, attention-free token importance measure.

Abstract

Large Vision-Language Models (LVLMs) have advanced multimodal learning but face high computational costs due to the large number of visual tokens, motivating token pruning to improve inference efficiency. The key challenge lies in identifying which tokens are truly important. Most existing approaches rely on attention-based criteria to estimate token importance. However, they inherently suffer from certain limitations, such as positional bias. In this work, we explore a new perspective on token importance based on token transitions in LVLMs. We observe that the transition of token representations provides a meaningful signal of semantic information. Based on this insight, we propose TransPrune, a training-free and efficient token pruning method. Specifically, TransPrune progressively prunes tokens by assessing their importance through a combination of Token Transition Variation (TTV)-which measures changes in both the magnitude and direction of token representations-and Instruction-Guided Attention (IGA), which measures how strongly the instruction attends to image tokens via attention. Extensive experiments demonstrate that TransPrune achieves comparable multimodal performance to original LVLMs, such as LLaVA-v1.5 and LLaVA-Next, across eight benchmarks, while reducing inference TFLOPs by more than half. Moreover, TTV alone can serve as an effective criterion without relying on attention, achieving performance comparable to attention-based methods. The code will be made publicly available upon acceptance of the paper at https://github.com/liaolea/TransPrune.