Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives

TL;DR

This paper reviews how machine learning models integrate structured and unstructured data from electronic health records to improve healthcare analysis, highlighting current methods, challenges, and future research directions.

Contribution

It provides a comprehensive overview of multimodal data fusion techniques in EHR-based machine learning research, emphasizing recent advances and identifying key challenges.

Findings

Multimodal data fusion enhances EHR analysis accuracy.

Deep learning models effectively combine structured and unstructured data.

Current methods face limitations in data integration and model interpretability.

Abstract

Background: Electronic Health Records (EHRs) contain rich information of patients' health history, which usually include both structured and unstructured data. There have been many studies focusing on distilling valuable information from structured data, such as disease codes, laboratory test results, and treatments. However, relying on structured data only might be insufficient in reflecting patients' comprehensive information and such data may occasionally contain erroneous records. Objective: With the recent advances of machine learning (ML) and deep learning (DL) techniques, an increasing number of studies seek to obtain more accurate results by incorporating unstructured free-text data as well. This paper reviews studies that use multimodal data, i.e. a combination of structured and unstructured data, from EHRs as input for conventional ML or DL models to address the targeted tasks. Materials and Methods: We searched in the Institute of Electrical and Electronics Engineers (IEEE) Digital Library, PubMed, and Association for Computing Machinery (ACM) Digital Library for articles related to ML-based multimodal EHR studies. Results and Discussion: With the final 94 included studies, we focus on how data from different modalities were combined and interacted using conventional ML and DL techniques, and how these algorithms were applied in EHR-related tasks. Further, we investigate the advantages and limitations of these fusion methods and indicate future directions for ML-based multimodal EHR research.