LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models

TL;DR

LVPruning is a straightforward language-guided method that prunes up to 90% of vision tokens in multi-modal large language models, significantly reducing computational costs with minimal performance loss.

Contribution

We propose LVPruning, a simple, parameter-free approach that uses cross-attention to prune vision tokens based on language interaction, improving efficiency in multi-modal models.

Findings

Reduces up to 90% of vision tokens in LLaVA-1.5

Decreases inference TFLOPs by 62.1%

Maintains 99.55% of original performance

Abstract

Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.