MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization

TL;DR

MOVER introduces a multimodal learning framework combining optimal transport and volume-based regularization to achieve semantically aligned, structured representations across multiple modalities, improving retrieval and generalization.

Contribution

It presents a novel combination of optimal transport and geometric regularization for multimodal embedding, enhancing cross-modal alignment and structure.

Findings

Outperforms state-of-the-art in text-video-audio retrieval

Improves zero-shot and finetuned retrieval performance

Enhances structural consistency and generalization in embeddings

Abstract

Recent advances in multimodal learning have largely relied on pairwise contrastive objectives to align different modalities, such as text, video, and audio, in a shared embedding space. While effective in bi-modal setups, these approaches struggle to generalize across multiple modalities and often lack semantic structure in high-dimensional spaces. In this paper, we propose MOVER, a novel framework that combines optimal transport-based soft alignment with volume-based geometric regularization to build semantically aligned and structured multimodal representations. By integrating a transport-guided matching mechanism with a geometric volume minimization objective (GAVE), MOVER encourages consistent alignment across all modalities in a modality-agnostic manner. Experiments on text-video-audio retrieval tasks demonstrate that MOVER significantly outperforms prior state-of-the-art methods in both zero-shot and finetuned settings. Additional analysis shows improved generalization to unseen modality combinations and stronger structural consistency in the learned embedding space.