SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

TL;DR

SCOPE introduces a novel token pruning method for multimodal large language models that jointly models saliency and coverage to better preserve semantic information while reducing computational costs.

Contribution

The paper proposes SCOPE, a new token pruning strategy that combines saliency and coverage modeling, improving semantic preservation in multimodal LLMs.

Findings

Outperforms prior token pruning methods on multiple benchmarks.

Effectively reduces computational overhead without sacrificing accuracy.

Demonstrates robustness across different vision-language tasks.

Abstract

Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called \textbf{S}aliency-\textbf{C}overage \textbf{O}riented token \textbf{P}runing for \textbf{E}fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches. Our code is available at \href{https://github.com/kinredon/SCOPE}{https://github.com/kinredon/SCOPE}.