The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment

TL;DR

The paper introduces Contrastive Fusion (ConFu), a novel framework for higher-order multimodal alignment that jointly embeds individual modalities and their fused combinations to better capture complex dependencies.

Contribution

ConFu extends contrastive learning to include fused modality representations, enabling higher-order dependency modeling while maintaining pairwise relationships.

Findings

ConFu captures higher-order dependencies like XOR relationships.

It demonstrates competitive performance on retrieval and classification tasks.

Supports unified one-to-one and two-to-one retrieval within a single framework.

Abstract

Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework. We release our code and dataset at https://github.com/estafons/confu.