Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models

TL;DR

Adaptive-VoCo introduces a dynamic visual token compression method for vision-language models, balancing computational efficiency and performance by adapting to image complexity during inference.

Contribution

It proposes a lightweight predictor for adaptive compression rates based on visual complexity, improving over fixed-rate methods in multimodal tasks.

Findings

Outperforms fixed-rate baselines across multiple tasks

Reduces computational costs while maintaining accuracy

Balances inference efficiency with representational capacity

Abstract

In recent years, large-scale vision-language models (VLMs) have demonstrated remarkable performance on multimodal understanding and reasoning tasks. However, handling high-dimensional visual features often incurs substantial computational and memory costs. VoCo-LLaMA alleviates this issue by compressing visual patch tokens into a few VoCo tokens, reducing computational overhead while preserving strong cross-modal alignment. Nevertheless, such approaches typically adopt a fixed compression rate, limiting their ability to adapt to varying levels of visual complexity. To address this limitation, we propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression. This predictor dynamically selects an optimal compression rate by quantifying an image's visual complexity using statistical cues from the vision encoder, such as patch token entropy and attention map variance. Furthermore, we introduce a joint loss function that integrates rate regularization with complexity alignment. This enables the model to balance inference efficiency with representational capacity, particularly in challenging scenarios. Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks, highlighting the potential of adaptive visual compression for creating more efficient and robust VLMs.