Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval

TL;DR

This paper introduces Reasoning Guided Embeddings (RGE), a novel approach that leverages the reasoning capabilities of Multimodal Large Language Models to produce more effective multimodal embeddings for retrieval tasks.

Contribution

The paper proposes RGE, a method that incorporates reasoning into embedding extraction, improving multimodal retrieval performance by explicitly leveraging MLLMs' generative reasoning.

Findings

RGE improves retrieval performance by 4.9% over baseline.

Explicit reasoning enhances embedding quality.

Structured rationale generation benefits multimodal tasks.

Abstract

Multimodal embeddings are widely used in downstream tasks such as multimodal retrieval, enabling alignment of interleaved modalities in a shared representation space. While recent studies show that Multimodal Large Language Models (MLLMs) can serve as strong embedding extractors, existing approaches treat embedding extraction as a direct encoding step, overlooking the fact that MLLMs possess the generative capability for reasoning that could be leveraged to enhance representation quality. In this work, we explore how to explicitly incorporate reasoning into the embedding process. To this end, we propose Reasoning Guided Embeddings (RGE), which preserves the generative rationale process of MLLMs and couples it with contrastive training. Our method first enables the model to perform structured rationale generation conditioned on the instruction, and then extracts representations after reasoning has unfolded. This simple design enhances the context-conditional inference signals within the embedding, leading to improved multimodal representation quality. Experiments on the MMEB benchmark show that reasoning-guided conditioning improves multimodal retrieval performance by 4.9% over the non-reasoning baseline, confirming that explicit reasoning can effectively enhance embedding quality.