Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

TL;DR

This review explores the development, challenges, and clinical applications of multimodal AI in medicine, highlighting its superior performance over unimodal models and discussing strategies for effective implementation.

Contribution

It provides a comprehensive overview of deep learning-based multimodal AI applications in healthcare, analyzing 432 papers and identifying key challenges and future directions.

Findings

Multimodal AI models outperform unimodal models by 6.2% in AUC.

Challenges include data heterogeneity and incomplete datasets.

Commercial multimodal AI models are emerging for clinical use.

Abstract

Recent technological advances in healthcare have led to unprecedented growth in patient data quantity and diversity. While artificial intelligence (AI) models have shown promising results in analyzing individual data modalities, there is increasing recognition that models integrating multiple complementary data sources, so-called multimodal AI, could enhance clinical decision-making. This scoping review examines the landscape of deep learning-based multimodal AI applications across the medical domain, analyzing 432 papers published between 2018 and 2024. We provide an extensive overview of multimodal AI development across different medical disciplines, examining various architectural approaches, fusion strategies, and common application areas. Our analysis reveals that multimodal AI models consistently outperform their unimodal counterparts, with an average improvement of 6.2 percentage points in AUC. However, several challenges persist, including cross-departmental coordination, heterogeneous data characteristics, and incomplete datasets. We critically assess the technical and practical challenges in developing multimodal AI systems and discuss potential strategies for their clinical implementation, including a brief overview of commercially available multimodal AI models for clinical decision-making. Additionally, we identify key factors driving multimodal AI development and propose recommendations to accelerate the field's maturation. This review provides researchers and clinicians with a thorough understanding of the current state, challenges, and future directions of multimodal AI in medicine.