Integrated Analysis for Electronic Health Records with Structured and Sporadic Missingness

TL;DR

This paper introduces Macomss, a novel imputation method designed for electronic health records with complex missing data patterns, improving data integration and analysis accuracy in healthcare research.

Contribution

The paper presents Macomss, a new imputation framework with theoretical guarantees, tailored for structured and sporadic missingness in EHR data, validated through simulations and real-world datasets.

Findings

Outperforms existing imputation methods in simulations

Achieves lowest imputation errors in DUHS datasets

Provides superior or comparable downstream prediction performance

Abstract

Objectives: We propose a novel imputation method tailored for Electronic Health Records (EHRs) with structured and sporadic missingness. Such missingness frequently arises in the integration of heterogeneous EHR datasets for downstream clinical applications. By addressing these gaps, our method provides a practical solution for integrated analysis, enhancing data utility and advancing the understanding of population health. Materials and Methods: We begin by demonstrating structured and sporadic missing mechanisms in the integrated analysis of EHR data. Following this, we introduce a novel imputation framework, Macomss, specifically designed to handle structurally and heterogeneously occurring missing data. We establish theoretical guarantees for Macomss, ensuring its robustness in preserving the integrity and reliability of integrated analyses. To assess its empirical performance, we conduct extensive simulation studies that replicate the complex missingness patterns observed in real-world EHR systems, complemented by validation using EHR datasets from the Duke University Health System (DUHS). Results: Simulation studies show that our approach consistently outperforms existing imputation methods. Using datasets from three hospitals within DUHS, Macomss achieves the lowest imputation errors for missing data in most cases and provides superior or comparable downstream prediction performance compared to benchmark methods. Conclusions: We provide a theoretically guaranteed and practically meaningful method for imputing structured and sporadic missing data, enabling accurate and reliable integrated analysis across multiple EHR datasets. The proposed approach holds significant potential for advancing research in population health.