FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records

TL;DR

This paper introduces a comprehensive evaluation framework for foundation models applied to structured electronic health records, assessing their clinical utility across diverse tasks and patient subpopulations.

Contribution

It proposes a suite of clinically meaningful tasks and benchmarks to systematically evaluate EHR foundation models' performance and robustness.

Findings

Foundation models show varied performance across tasks.

Calibration and subpopulation performance reveal tradeoffs.

Pre-training and data strategies significantly impact results.

Abstract

Foundation models hold significant promise in healthcare, given their capacity to extract meaningful representations independent of downstream tasks. This property has enabled state-of-the-art performance across several clinical applications trained on structured electronic health record (EHR) data, even in settings with limited labeled data, a prevalent challenge in healthcare. However, there is little consensus on these models' potential for clinical utility due to the lack of desiderata of comprehensive and meaningful tasks and sufficiently diverse evaluations to characterize the benefit over conventional supervised learning. To address this gap, we propose a suite of clinically meaningful tasks spanning patient outcomes, early prediction of acute and chronic conditions, including desiderata for robust evaluations. We evaluate state-of-the-art foundation models on EHR data consisting of 5 million patients from Columbia University Irving Medical Center (CUMC), a large urban academic medical center in New York City, across 14 clinically relevant tasks. We measure overall accuracy, calibration, and subpopulation performance to surface tradeoffs based on the choice of pre-training, tokenization, and data representation strategies. Our study aims to advance the empirical evaluation of structured EHR foundation models and guide the development of future healthcare foundation models.