Robust Multimodal Learning via Representation Decoupling

TL;DR

This paper introduces DMRNet, a novel multimodal learning model that models input as probabilistic distributions to better capture modality-specific information and improve robustness to missing modalities.

Contribution

The paper proposes DMRNet, which models multimodal inputs as distributions and uses sampling to relax constraints, enhancing modality-specific learning and robustness.

Findings

DMRNet outperforms state-of-the-art methods on classification tasks.

The probabilistic approach improves robustness to missing modalities.

The hard combination regularizer balances training across modality combinations.

Abstract

Multimodal learning robust to missing modality has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. However, we reveal that they are sub-optimal due to their implicit constraint on intra-class representation. Specifically, the sample with different modalities within the same class will be forced to learn representations in the same direction. This hinders the model from capturing modality-specific information, resulting in insufficient learning. To this end, we propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning. Specifically, DMRNet models the input from different modality combinations as a probabilistic distribution instead of a fixed point in the latent space, and samples embeddings from the distribution for the prediction module to calculate the task loss. As a result, the direction constraint from the loss minimization is blocked by the sampled representation. This relaxes the constraint on the inference representation and enables the model to capture the specific information for different modality combinations. Furthermore, we introduce a hard combination regularizer to prevent DMRNet from unbalanced training by guiding it to pay more attention to hard modality combinations. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that the proposed DMRNet outperforms the state-of-the-art significantly.