mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data

TL;DR

This paper introduces mmE5, a multimodal multilingual embedding model trained on high-quality synthetic data that covers diverse tasks and modalities, leading to state-of-the-art results in multimodal and multilingual benchmarks.

Contribution

The work presents a novel data synthesis approach guided by three quality criteria, enabling the creation of diverse, aligned, and realistic synthetic datasets for training multimodal multilingual models.

Findings

mmE5 achieves state-of-the-art results on MMEB benchmark.

Synthetic data quality significantly impacts embedding performance.

High-quality synthetic datasets improve multilingual multimodal understanding.

Abstract

Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space. However, the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck. In this work, we identify three criteria for high-quality synthetic multimodal data. First, broad scope ensures that the generated data covers diverse tasks and modalities, making it applicable to various downstream scenarios. Second, robust cross-modal alignment makes different modalities semantically consistent. Third, high fidelity ensures that the synthetic data maintains realistic details to enhance its reliability. Guided by these principles, we synthesize datasets that: (1) cover a wide range of tasks, modality combinations, and languages, (2) are generated via a deep thinking process within a single pass of a multimodal large language model, and (3) incorporate real-world images with accurate and relevant texts, ensuring fidelity through self-evaluation and refinement. Leveraging these high-quality synthetic and labeled datasets, we train a multimodal multilingual E5 model mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art performance on the MMEB Benchmark and superior multilingual performance on the XTD benchmark. Our codes, datasets and models are released in https://github.com/haon-chen/mmE5.