LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models

TL;DR

LaCo is a novel layer-wise visual token compression framework for multimodal large language models that improves efficiency and maintains performance by compressing tokens within the vision encoder layers.

Contribution

LaCo introduces a new layer-wise compression method with pixel-shuffle and residual learning, enabling more effective and efficient token compression during model training and inference.

Findings

Outperforms existing token compression methods in effectiveness.

Improves training efficiency by over 20%.

Enhances inference throughput by over 15%.

Abstract

Existing visual token compression methods for Multimodal Large Language Models (MLLMs) predominantly operate as post-encoder modules, limiting their potential for efficiency gains. To address this limitation, we propose LaCo (Layer-wise Visual Token Compression), a novel framework that enables effective token compression within the intermediate layers of the vision encoder. LaCo introduces two core components: 1) a layer-wise pixel-shuffle mechanism that systematically merges adjacent tokens through space-to-channel transformations, and 2) a residual learning architecture with non-parametric shortcuts that preserves critical visual information during compression. Extensive experiments indicate that our LaCo outperforms all existing methods when compressing tokens in the intermediate layers of the vision encoder, demonstrating superior effectiveness. In addition, compared to external compression, our method improves training efficiency beyond 20% and inference throughput over 15% while maintaining strong performance.