Automated Ensemble Multimodal Machine Learning for Healthcare

TL;DR

This paper introduces AutoPrognosis-M, an automated multimodal machine learning framework that integrates clinical data and medical imaging to improve healthcare diagnostics, demonstrating the value of combining multiple data sources and fusion strategies.

Contribution

The paper presents a novel automated framework for multimodal healthcare data integration, including multiple imaging models and fusion strategies, facilitating clinical adoption and open-sourcing the tool.

Findings

Multimodal learning improves diagnostic accuracy.

Combining multiple fusion strategies enhances model robustness.

Open-source framework accelerates multimodal ML adoption in healthcare.

Abstract

The application of machine learning in medicine and healthcare has led to the creation of numerous diagnostic and prognostic models. However, despite their success, current approaches generally issue predictions using data from a single modality. This stands in stark contrast with clinician decision-making which employs diverse information from multiple sources. While several multimodal machine learning approaches exist, significant challenges in developing multimodal systems remain that are hindering clinical adoption. In this paper, we introduce a multimodal framework, AutoPrognosis-M, that enables the integration of structured clinical (tabular) data and medical imaging using automated machine learning. AutoPrognosis-M incorporates 17 imaging models, including convolutional neural networks and vision transformers, and three distinct multimodal fusion strategies. In an illustrative application using a multimodal skin lesion dataset, we highlight the importance of multimodal machine learning and the power of combining multiple fusion strategies using ensemble learning. We have open-sourced our framework as a tool for the community and hope it will accelerate the uptake of multimodal machine learning in healthcare and spur further innovation.