AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs

TL;DR

AdaTok introduces an object-aware token merging strategy for multimodal large language models, significantly reducing token count and computational load while maintaining high performance, inspired by human vision cognition.

Contribution

The paper presents a novel object-level token merging method for adaptive token compression in MLLMs, aligning with human vision and improving efficiency without sacrificing accuracy.

Findings

Uses only 10% of tokens compared to vanilla models

Achieves 96% of the original model's performance

Outperforms relevant methods in balancing compression and accuracy

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available.