VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference

TL;DR

VISA introduces a graph-based group-wise visual token selection and aggregation method that efficiently compresses visual tokens in multimoal large language models, improving inference speed while preserving more visual information.

Contribution

The paper proposes a novel graph-based visual token aggregation and a group-wise token selection strategy to enhance efficiency and information retention in MLLMs.

Findings

Outperforms previous token pruning methods in benchmarks.

Achieves better trade-off between performance and inference speed.

Validated on multiple datasets including LLaVA-1.5 and Video-LLaVA.

Abstract

In this study, we introduce a novel method called group-wise \textbf{VI}sual token \textbf{S}election and \textbf{A}ggregation (VISA) to address the issue of inefficient inference stemming from excessive visual tokens in multimoal large language models (MLLMs). Compared with previous token pruning approaches, our method can preserve more visual information while compressing visual tokens. We first propose a graph-based visual token aggregation (VTA) module. VTA treats each visual token as a node, forming a graph based on semantic similarity among visual tokens. It then aggregates information from removed tokens into kept tokens based on this graph, producing a more compact visual token representation. Additionally, we introduce a group-wise token selection strategy (GTS) to divide visual tokens into kept and removed ones, guided by text tokens from the final layers of each group. This strategy progressively aggregates visual information, enhancing the stability of the visual information extraction process. We conduct comprehensive experiments on LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA across various benchmarks to validate the efficacy of VISA. Our method consistently outperforms previous methods, achieving a superior trade-off between model performance and inference speed. The code is available at https://github.com/mobiushy/VISA.