RzenEmbed: Towards Comprehensive Multimodal Retrieval

TL;DR

RzenEmbed is a unified multimodal embedding framework that enhances retrieval across diverse visual and textual modalities by employing a novel training strategy and improved loss functions, setting new state-of-the-art results.

Contribution

The paper introduces RzenEmbed, a comprehensive multimodal embedding model with a two-stage training process and advanced loss techniques, supporting a wider range of modalities than prior methods.

Findings

Achieves state-of-the-art on MMEB benchmark

Outperforms previous models in video retrieval

Improves instruction-following capabilities

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has extended CLIP-based frameworks to produce powerful, universal embeddings for retrieval tasks. However, existing methods primarily focus on natural images, offering limited support for other crucial visual modalities such as videos and visual documents. To bridge this gap, we introduce RzenEmbed, a unified framework to learn embeddings across a diverse set of modalities, including text, images, videos, and visual documents. We employ a novel two-stage training strategy to learn discriminative representations. The first stage focuses on foundational text and multimodal retrieval. In the second stage, we introduce an improved InfoNCE loss, incorporating two key enhancements. Firstly, a hardness-weighted mechanism guides the model to prioritize challenging samples by assigning them higher weights within each batch. Secondly, we implement an approach to mitigate the impact of false negatives and alleviate data noise. This strategy not only enhances the model's discriminative power but also improves its instruction-following capabilities. We further boost performance with learnable temperature parameter and model souping. RzenEmbed sets a new state-of-the-art on the MMEB benchmark. It not only achieves the best overall score but also outperforms all prior work on the challenging video and visual document retrieval tasks. Our models are available in https://huggingface.co/qihoo360/RzenEmbed.