ReMatch: Boosting Representation through Matching for Multimodal Retrieval

TL;DR

ReMatch introduces a novel multimodal retrieval framework that leverages the generative capabilities of MLLMs through end-to-end training and generative matching, achieving state-of-the-art results and strong zero-shot generalization.

Contribution

It proposes a new training approach that utilizes generative matching with MLLMs for improved multimodal retrieval performance.

Findings

Achieves new state-of-the-art on MMEB benchmark.

Demonstrates strong zero-shot generalization across five datasets.

Utilizes multiple learnable tokens for richer embeddings.

Abstract

We present ReMatch, a framework that leverages the generative strength of MLLMs for multimodal retrieval. Previous approaches treated an MLLM as a simple encoder, ignoring its generative nature, and under-utilising its compositional reasoning and world knowledge. We instead train the embedding MLLM end-to-end with a chat-style generative matching stage. The matching stage uses the same MLLM to autoregressively decide relevance from multi-view inputs, including both raw data and its own projected embeddings for each query and document. It provides instance-wise discrimination supervision that complements a standard contrastive loss, offering stronger gradients on hard negatives and preserving the compositional strengths of the original MLLM. To obtain semantically richer multimodal embeddings, we use multiple learnable tokens to augment each input, generating fine-grained contextual, mutually orthogonal embeddings with low inference cost. Leveraging our established high-performance baseline,we assemble the ideas mentioned above into a powerful training recipe and achieve a new state-of-the-art on the Massive Multimodal Embedding Benchmark (MMEB). Our experiments show particularly strong zero-shot generalization results on five datasets, highlighting the robustness and transferability of ReMatch.