Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

TL;DR

COrAL is a novel multimodal contrastive learning framework that explicitly disentangles shared, unique, and synergistic information using orthogonality constraints and asymmetric masking, leading to more stable and comprehensive representations.

Contribution

The paper introduces COrAL, a framework that explicitly models all information components in multimodal data, improving representation quality and stability over existing methods.

Findings

Outperforms state-of-the-art on synthetic and real datasets

Achieves lower variance in performance across runs

Produces more stable and comprehensive multimodal embeddings

Abstract

Multimodal learning seeks to integrate information from heterogeneous sources, where signals may be shared across modalities, specific to individual modalities, or emerge only through their interaction. While self-supervised multimodal contrastive learning has achieved remarkable progress, most existing methods predominantly capture redundant cross-modal signals, often neglecting modality-specific (unique) and interaction-driven (synergistic) information. Recent extensions broaden this perspective, yet they either fail to explicitly model synergistic interactions or learn different information components in an entangled manner, leading to incomplete representations and potential information leakage. We introduce \textbf{COrAL}, a principled framework that explicitly and simultaneously preserves redundant, unique, and synergistic information within multimodal representations. COrAL employs a dual-path architecture with orthogonality constraints to disentangle shared and modality-specific features, ensuring a clean separation of information components. To promote synergy modeling, we introduce asymmetric masking with complementary view-specific patterns, compelling the model to infer cross-modal dependencies rather than rely solely on redundant cues. Extensive experiments on synthetic benchmarks and diverse MultiBench datasets demonstrate that COrAL consistently matches or outperforms state-of-the-art methods while exhibiting low performance variance across runs. These results indicate that explicitly modeling the full spectrum of multimodal information yields more stable, reliable, and comprehensive embeddings.