Bottleneck Tokens for Unified Multimodal Retrieval

TL;DR

This paper introduces Bottleneck Tokens and Generative Information Condensation to improve unified multimodal retrieval in decoder-only large language models, achieving state-of-the-art results on diverse datasets.

Contribution

It proposes a novel explicit pooling mechanism with learnable bottleneck tokens and a training method that enhances semantic compression for multimodal retrieval.

Findings

Achieves state-of-the-art performance on MMEB-V2 with 78 datasets and 3 modalities.

Substantial improvements on semantically demanding tasks like Video-QA.

Efficient inference with negligible overhead over traditional pooling methods.

Abstract

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).