Deep Metric Loss for Multimodal Learning

TL;DR

This paper introduces the MultiModal loss, a novel function that improves multimodal learning by subgrouping instances based on unimodal contributions, leading to better performance and more reliable modality-specific scores.

Contribution

The paper proposes the MultiModal loss, a new loss paradigm that accounts for varying unimodal contributions and enhances multimodal model optimization.

Findings

Improved classification performance on synthetic data.

Enhanced accuracy on four real multimodal datasets.

Generated reliable modality-specific prediction scores.

Abstract

Multimodal learning often outperforms its unimodal counterparts by exploiting unimodal contributions and cross-modal interactions. However, focusing only on integrating multimodal features into a unified comprehensive representation overlooks the unimodal characteristics. In real data, the contributions of modalities can vary from instance to instance, and they often reinforce or conflict with each other. In this study, we introduce a novel \text{MultiModal} loss paradigm for multimodal learning, which subgroups instances according to their unimodal contributions. \text{MultiModal} loss can prevent inefficient learning caused by overfitting and efficiently optimize multimodal models. On synthetic data, \text{MultiModal} loss demonstrates improved classification performance by subgrouping difficult instances within certain modalities. On four real multimodal datasets, our loss is empirically shown to improve the performance of recent models. Ablation studies verify the effectiveness of our loss. Additionally, we show that our loss generates a reliable prediction score for each modality, which is essential for subgrouping. Our \text{MultiModal} loss is a novel loss function to subgroup instances according to the contribution of modalities in multimodal learning and is applicable to a variety of multimodal models with unimodal decisions. Our code is available at https://github.com/SehwanMoon/MultiModalLoss.