MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

TL;DR

This paper introduces MMEB-V3, a comprehensive benchmark for evaluating full-modality embeddings across text, images, videos, and audio, revealing current models' limitations in modality-aware retrieval.

Contribution

The work presents MMEB-V3 and OmniSET, enabling systematic evaluation and diagnosis of full-modality embedding models, addressing a gap in existing benchmarks.

Findings

Models often fail to retrieve the target modality.

Cross-modal retrieval is highly asymmetric and query-modality biased.

Instruction-induced shifts are insufficient or misaligned with target modalities.

Abstract

Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.