Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

TL;DR

This systematic study evaluates pretraining strategies and dataset scaling effects on ECG foundation models, highlighting the superiority of structured state space models and contrastive predictive coding for diverse clinical tasks.

Contribution

It provides a comprehensive comparison of pretraining methods, architectures, and dataset sizes for ECG models, emphasizing the importance of model inductive biases.

Findings

Contrastive predictive coding outperforms other objectives in transferability.

Scaling data up to 11M samples improves model performance.

Structured state space models outperform transformers and CNNs.

Abstract

Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the size of the pretraining dataset are rarely assessed systematically and in a like-for-like manner. This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide. We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models, and investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources. Pretraining strategy has a meaningful and consistent impact on downstream performance, with contrastive predictive coding (slightly ahead of JEPA) yielding the most transferable representations across diverse clinical tasks. Scaling pretraining data continues to yield meaningful improvements up to 11M samples for most objectives. We also compare model architectures across all pretraining methodologies and find evidence for a clear superiority of structured state space models compared to transformers and CNN models. We hypothesize that the strong inductive biases of structured state space models, rather than pretraining scale alone, are the primary driver of effective ECG representation learning, with important implications for future foundation model development in this and potentially other physiological signal domains.