ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance

TL;DR

ToDRE introduces a novel, training-free visual token pruning method that enhances efficiency by separately considering token diversity and task relevance, achieving significant speed-ups with minimal performance loss.

Contribution

The paper proposes a two-stage, training-free framework that effectively prunes visual tokens by leveraging token diversity and task relevance, improving inference speed in large vision-language models.

Findings

Prunes 90% of visual tokens after the vision encoder.

Achieves 2.6x inference speed-up while maintaining 95% of model performance.

Effectively prunes all visual tokens in certain LLM decoder layers.

Abstract

Visual token pruning aims to compress and prune redundant visual tokens which play a critical role in efficient inference with large vision-language models (LVLMs). However, most existing work estimates visual redundancy using a single metric, such as cross-modal attention or visual token similarity. We show that visual token diversity and task-specific token relevance are two crucial yet orthogonal factors that complement each other in conveying useful information and should therefore be treated separately for more effective visual token pruning. Building upon this insight, we design TODRE, a two-stage and training-free framework that incorporates Token Diversity and task RElevance for effective token compression and efficient LVLM inference. Instead of pruning redundant tokens, we introduce a greedy max-sum diversification algorithm that selects and retains a subset of diverse and representative visual tokens after the vision encoder. On top of that, ToDRE leverages an "information migration" mechanism to eliminate task-irrelevant visual tokens within certain decoder layers of large language model(LLM) to further improve token pruning and LVLM inference. Extensive experiments show that ToDRE prunes 90% of visual tokens after the vision encoder as well as all visual tokens in certain LLM decoder layers, leading to a 2.6x speed-up in total inference time while maintaining 95.0% model performance plus excellent model compatibility.