Redundancy-Adaptive Multimodal Learning for Imperfect Data

TL;DR

This paper introduces RAML, a novel multimodal learning approach that effectively utilizes redundancy across modalities to improve robustness against imperfect data, outperforming existing methods.

Contribution

RAML is a new method that exploits unimodal redundancy and enforces feature constraints to enhance robustness in multimodal learning with imperfect data.

Findings

RAML outperforms state-of-the-art methods on benchmark datasets.

RAML effectively handles corrupted and missing modality data.

RAML maintains high performance with complete modality data.

Abstract

Multimodal models trained on complete modality data often exhibit a substantial decrease in performance when faced with imperfect data containing corruptions or missing modalities. To address this robustness challenge, prior methods have explored various approaches from aspects of augmentation, consistency or uncertainty, but these approaches come with associated drawbacks related to data complexity, representation, and learning, potentially diminishing their overall effectiveness. In response to these challenges, this study introduces a novel approach known as the Redundancy-Adaptive Multimodal Learning (RAML). RAML efficiently harnesses information redundancy across multiple modalities to combat the issues posed by imperfect data while remaining compatible with the complete modality. Specifically, RAML achieves redundancy-lossless information extraction through separate unimodal discriminative tasks and enforces a proper norm constraint on each unimodal feature representation. Furthermore, RAML explicitly enhances multimodal fusion by leveraging fine-grained redundancy among unimodal features to learn correspondences between corrupted and untainted information. Extensive experiments on various benchmark datasets under diverse conditions have consistently demonstrated that RAML outperforms state-of-the-art methods by a significant margin.