When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion

TL;DR

This paper systematically benchmarks multimodal fusion of EHR and chest X-ray data in healthcare, revealing when it improves predictions, its robustness to missing data, and its impact on fairness, supported by an open-source toolkit.

Contribution

It provides a comprehensive benchmark and analysis of multimodal fusion strategies in healthcare, highlighting conditions for effectiveness and limitations, and introduces a flexible evaluation toolkit.

Findings

Multimodal fusion improves performance with complete data, especially for diseases needing both modalities.

Rich temporal EHR data introduces modality imbalance that complex architectures can't fully address.

Benefits of multimodal fusion decline with missing data unless models are designed for incompleteness.

Abstract

Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.