Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model

TL;DR

MustDrop is a multi-stage token dropping method for multimodal large language models that improves inference efficiency by accurately identifying and removing redundant vision tokens across encoding, prefilling, and decoding stages.

Contribution

It introduces a comprehensive multi-stage token importance measurement and filtering strategy that enhances efficiency without sacrificing accuracy in multimodal LLMs.

Findings

Reduces about 88.5% FLOPs on LLaVA.

Achieves a compression ratio of 92.2%.

Maintains comparable accuracy with full-token models.

Abstract

The vision tokens in multimodal large language models usually exhibit significant spatial and temporal redundancy and take up most of the input tokens, which harms their inference efficiency. To solve this problem, some recent works were introduced to drop the unimportant tokens during inference where the importance of each token is decided only by the information in either the vision encoding stage or the prefilling stage. In this paper, we propose Multi-stage Token Dropping (MustDrop) to measure the importance of each token from the whole lifecycle, including the vision encoding stage, prefilling stage, and decoding stage. Concretely, in the visual encoding stage, MustDrop merges spatially adjacent tokens with high similarity, and establishes a key token set to retain the most vision-critical tokens, preventing them from being discarded in later stages. In the prefilling stage, MustDrop further compresses vision tokens by the guidance of text semantics, with a dual-attention filtering strategy. In the decoding stage, an output-aware cache policy is proposed to further reduce the size of the KV cache. By leveraging tailored strategies in the multi-stage process, MustDrop can more precisely recognize the important and redundant tokens, thus achieving an optimal balance between performance and efficiency. For instance, MustDrop reduces about 88.5\% FLOPs on LLaVA with a compression ratio of 92.2\% while maintaining comparable accuracy. Our codes are available at \url{https://github.com/liuting20/MustDrop}.