Towards Uniformity and Alignment for Multimodal Representation Learning

TL;DR

This paper identifies conflicts in multimodal representation learning caused by InfoNCE objectives and proposes a decoupled approach to improve alignment and uniformity, leading to better cross-modal tasks.

Contribution

It introduces a novel decoupling method for alignment and uniformity in multimodal learning, backed by theoretical guarantees and extensive experimental validation.

Findings

Reduces distribution gaps across modalities.

Improves retrieval and generation performance.

Provides theoretical guarantees for the method.

Abstract

Multimodal representation learning aims to construct a shared embedding space in which heterogeneous modalities are semantically aligned. Despite strong empirical results, InfoNCE-based objectives introduce inherent conflicts that yield distribution gaps across modalities. In this work, we identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases: (i) an alignment-uniformity conflict, whereby the repulsion of uniformity undermines pairwise alignment, and (ii) an intra-alignment conflict, where aligning multiple modalities induces competing alignment directions. To address these issues, we propose a principled decoupling of alignment and uniformity for multimodal representations, providing a conflict-free recipe for multimodal learning that simultaneously supports discriminative and generative use cases without task-specific modules. We then provide a theoretical guarantee that our method acts as an efficient proxy for a global H\"older divergence over multiple modality distributions, and thus reduces the distribution gap among modalities. Extensive experiments on retrieval and UnCLIP-style generation demonstrate consistent gains.