Deep Multimodal Learning with Missing Modality: A Survey

TL;DR

This survey reviews recent deep learning approaches for multimodal learning when some data modalities are missing, highlighting methods, applications, datasets, challenges, and future directions to improve model robustness.

Contribution

It provides the first comprehensive overview of deep multimodal learning with missing modalities, clarifying its motivation, distinctions, and current research landscape.

Findings

Summarizes recent deep learning methods for MLMM

Analyzes applications and datasets in MLMM

Discusses challenges and future research directions

Abstract

During multimodal model training and testing, certain data modalities may be absent due to sensor limitations, cost constraints, privacy concerns, or data loss, negatively affecting performance. Multimodal learning techniques designed to handle missing modalities can mitigate this by ensuring model robustness even when some modalities are unavailable. This survey reviews recent progress in Multimodal Learning with Missing Modality (MLMM), focusing on deep learning methods. It provides the first comprehensive survey that covers the motivation and distinctions between MLMM and standard multimodal learning setups, followed by a detailed analysis of current methods, applications, and datasets, concluding with challenges and future directions.