Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

TL;DR

RAVEN is a recurrence-aware generative model for sequential electronic health records that predicts next visits, scales with data, and generalizes across cohorts, advancing healthcare foundation models.

Contribution

Introduces RAVEN, a novel pretraining strategy for EHRs that addresses repeated event inflation and explores scaling behaviors in data-constrained environments.

Findings

RAVEN rivals fine-tuned Transformer models in disease prediction.

Model generalizes to external cohorts without additional training.

Scaling model size without more data is suboptimal.

Abstract

While large-scale pretraining has revolutionized language modeling, its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction. Leveraging a dataset of over one million unique individuals, our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history. We introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Furthermore, we empirically investigate the scaling behaviors in a data-constrained, compute-saturated regime, showing that simply increasing model size is suboptimal without commensurate increases in data volume. We evaluate our model via zero-shot prediction for forecasting the incidence of a diverse set of diseases, where it rivals fully fine-tuned representation-based Transformer models and outperforms both standard simulation-based next-token approaches and a prompted medical large language model baseline. Finally, without additional parameter updates, we show that RAVEN can generalize to an external patient cohort under lossy clinical code mappings and feature coverage gaps.