Towards dynamic multi-modal phenotyping using chest radiographs and physiological data

TL;DR

This paper introduces a dynamic multi-modal learning approach combining chest radiographs and physiological data to improve patient phenotyping accuracy, demonstrating significant performance gains over single-modality models.

Contribution

It proposes a novel dynamic training method for integrating multi-modal data representations in medical phenotyping tasks, enhancing diagnostic performance.

Findings

Achieved highest AUROC of 0.764 using combined modalities.

Improved AUROC from 0.747 to 0.798 for chronic disease detection.

Demonstrated the benefit of multi-modal learning in healthcare applications.

Abstract

The healthcare domain is characterized by heterogeneous data modalities, such as imaging and physiological data. In practice, the variety of medical data assists clinicians in decision-making. However, most of the current state-of-the-art deep learning models solely rely upon carefully curated data of a single modality. In this paper, we propose a dynamic training approach to learn modality-specific data representations and to integrate auxiliary features, instead of solely relying on a single modality. Our preliminary experiments results for a patient phenotyping task using physiological data in MIMIC-IV & chest radiographs in the MIMIC- CXR dataset show that our proposed approach achieves the highest area under the receiver operating characteristic curve (AUROC) (0.764 AUROC) compared to the performance of the benchmark method in previous work, which only used physiological data (0.740 AUROC). For a set of five recurring or chronic diseases with periodic acute episodes, including cardiac dysrhythmia, conduction disorders, and congestive heart failure, the AUROC improves from 0.747 to 0.798. This illustrates the benefit of leveraging the chest imaging modality in the phenotyping task and highlights the potential of multi-modal learning in medical applications.