AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning

TL;DR

This paper introduces a training-free, adaptive inference technique for multi-modal LLMs that significantly reduces computational costs through token merging and pruning, while maintaining high performance across image and video benchmarks.

Contribution

It proposes a novel, minimalist, training-free method combining token merging and pruning to improve efficiency of multi-modal LLMs across diverse tasks.

Findings

7-fold reduction in FLOPs with minimal performance loss

Outperforms state-of-the-art methods in long video understanding

Provides insights into token redundancy and layer behaviors

Abstract

Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that our method substantially reduces computation load (e.g., a $\textbf{7-fold}$ reduction in FLOPs) while preserving the performance of video and image LLMs. Further, at a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., $\textbf{+4.6}$ on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code is available at https://github.com/LaVi-Lab/AIM.