Salvaging Forbidden Treasure in Medical Data: Utilizing Surrogate Outcomes and Single Records for Rare Event Modeling

TL;DR

This paper introduces a hybrid learning framework that leverages surrogate outcomes and single-record patient data from electronic health records to improve modeling of rare events like suicide attempts.

Contribution

It presents a novel integrative approach combining supervised and unsupervised learning to utilize otherwise discarded single-record data and surrogate outcomes for rare event prediction.

Findings

Single-record data contain valuable information for rare event modeling.

Using surrogate outcomes improves the accuracy of suicide risk prediction.

The method significantly outperforms traditional models that exclude single-record patients.

Abstract

The vast repositories of Electronic Health Records (EHR) and medical claims hold untapped potential for studying rare but critical events, such as suicide attempt. Conventional setups often model suicide attempt as a univariate outcome and also exclude any ``single-record'' patients with a single documented encounter due to a lack of historical information. However, patients who were diagnosed with suicide attempts at the only encounter could, to some surprise, represent a substantial proportion of all attempt cases in the data, as high as 70--80%. We innovate a hybrid and integrative learning framework to leverage concurrent outcomes as surrogates and harness the forbidden yet precious information from single-record data. Our approach employs a supervised learning component to learn the latent variables that connect primary (e.g., suicide) and surrogate outcomes (e.g., mental disorders) to historical information. It simultaneously employs an unsupervised learning component to utilize the single-record data, through the shared latent variables. As such, our approach offers a general strategy for information integration that is crucial to modeling rare conditions and events. With hospital inpatient data from Connecticut, we demonstrate that single-record data and concurrent diagnoses indeed carry valuable information, and utilizing them can substantially improve suicide risk modeling.