Exploring Scaling Laws for EHR Foundation Models

TL;DR

This paper investigates how scaling laws apply to EHR foundation models, revealing predictable performance patterns and guiding resource-efficient training for clinical applications.

Contribution

First empirical study demonstrating that EHR models follow scaling laws similar to language models, with insights for efficient development of clinical AI systems.

Findings

EHR models exhibit parabolic IsoFLOPs scaling curves.

Power-law relationships found between compute, model size, and data.

Scaling laws can predict clinical utility improvements.

Abstract

The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model size, dataset volume, and compute. Yet, these principles remain largely unexplored in the context of electronic health records (EHRs) -- a rich, sequential, and globally abundant data source that differs structurally from natural language. In this work, we present the first empirical investigation of scaling laws for EHR foundation models. By training transformer architectures on patient timeline data from the MIMIC-IV database across varying model sizes and compute budgets, we identify consistent scaling patterns, including parabolic IsoFLOPs curves and power-law relationships between compute, model parameters, data size, and clinical utility. These findings demonstrate that EHR models exhibit scaling behavior analogous to LLMs, offering predictive insights into resource-efficient training strategies. Our results lay the groundwork for developing powerful EHR foundation models capable of transforming clinical prediction tasks and advancing personalized healthcare.