Reliable Multimodal Learning Via Multi-Level Adaptive DeConfusion

TL;DR

This paper introduces MLAD, a novel method that reduces inter-class and sample-specific confusion in multimodal learning, significantly improving classification reliability especially in noisy data scenarios.

Contribution

MLAD is the first approach to eliminate inter-class confusion at both global and sample levels in multimodal learning, enhancing model reliability.

Findings

MLAD outperforms state-of-the-art methods on multiple benchmarks.

MLAD achieves higher classification confidence in noisy data.

MLAD demonstrates superior reliability in real-world scenarios.

Abstract

Multimodal learning enhances the performance of various machine learning tasks by leveraging complementary information across different modalities. However, existing methods often learn multimodal representations that retain substantial inter-class confusion, making it difficult to achieve high-confidence predictions, particularly in real-world scenarios with low-quality or noisy data. To address this challenge, we propose Multi-Level Adaptive DeConfusion (MLAD), which eliminates inter-class confusion in multimodal data at both global and sample levels, significantly enhancing the classification reliability of multimodal models. Specifically, MLAD first learns class-wise latent distributions with global-level confusion removed via dynamic-exit modality encoders that adapt to the varying discrimination difficulty of each class and a cross-class residual reconstruction mechanism. Subsequently, MLAD further removes sample-specific confusion through sample-adaptive cross-modality rectification guided by confusion-free modality priors. These priors are constructed from low-confusion modality features, identified by evaluating feature confusion using the learned class-wise latent distributions and selecting those with low confusion via a Gaussian mixture model. Experiments demonstrate that MLAD outperforms state-of-the-art methods across multiple benchmarks and exhibits superior reliability.