QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression

TL;DR

QMoP introduces an adaptive, query-guided framework for visual token compression in multimodal models, significantly reducing resource usage while maintaining performance through a multi-branch, dynamic selection approach.

Contribution

The paper presents QMoP, a novel flexible framework with a query-guided router and mixture-of-experts fusion for adaptive visual token compression in multimodal models.

Findings

QMoP achieves better compression-performance trade-offs than baselines.

The framework significantly reduces memory and computation costs.

VTCBench effectively evaluates information loss from compression.

Abstract

Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.