Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models

TL;DR

This paper introduces Attention-Driven Self-Compression (ADSC), a novel method for reducing vision tokens in multimodal large language models by leveraging the LLM's attention, significantly improving efficiency while maintaining high performance.

Contribution

The paper proposes a new token compression technique that uses the LLM's attention to guide progressive vision token reduction without auxiliary modules or attention modifications.

Findings

Reduces FLOPs by 53.7% and memory by 56.7% on LLaVA-1.5

Maintains 98.2% of original performance after compression

Outperforms prior pruning methods in efficiency and accuracy

Abstract

Multimodal Large Language Models (MLLMs) incur significant computational cost from processing numerous vision tokens through all LLM layers. Prior pruning methods operate either before the LLM, limiting generality due to diverse encoder-projector designs or within the LLM using heuristics that are incompatible with FlashAttention. We take a different approach: rather than identifying unimportant tokens, we treat the LLM itself as the optimal guide for compression. Observing that deeper layers naturally transmit vision-to-text information, we introduce Attention-Driven Self-Compression (ADSC), a simple, broadly applicable method that progressively reduces vision tokens using only the LLM's attention mechanism. Our method applies uniform token downsampling at selected layers, forming bottlenecks that encourage the model to reorganize and compress information into the remaining tokens. It requires no score computation, auxiliary modules, or attention modification, and remains fully compatible with FlashAttention. Applied to LLaVA-1.5, ADSC reduces FLOPs by 53.7% and peak KV-cache memory by 56.7%, while preserving 98.2% of the original model performance. Across multiple benchmarks, it outperforms prior pruning approaches in both efficiency and accuracy. Crucially, under high compression ratios, our method remains robust while heuristic-based techniques degrade sharply.