Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification

TL;DR

This paper introduces M3CoL, a multimodal mixup contrastive learning method that captures shared relations across modalities, improving classification performance across diverse datasets by aligning mixed samples and integrating auxiliary supervision.

Contribution

The paper proposes a novel mixup-based contrastive loss and a framework that captures shared relations in multimodal data, enhancing robustness and generalization in multimodal classification.

Findings

Outperforms state-of-the-art on N24News, ROSMAP, and BRCA datasets.

Achieves comparable performance on Food-101.

Effectively captures shared multimodal relations through mixup contrastive learning.

Abstract

Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach to capture nuanced shared relations inherent in multimodal data. Our key contribution is a Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with their corresponding samples from other modalities thereby capturing shared relations between them. For multimodal classification tasks, we introduce a framework that integrates a fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. It outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance on Food-101. Our work highlights the significance of learning shared relations for robust multimodal learning, opening up promising avenues for future research. Our code is publicly available at https://github.com/RaghavSinghal10/M3CoL.