OSF: On Pre-training and Scaling of Sleep Foundation Models

TL;DR

This paper introduces OSF, a family of sleep foundation models trained on a large, diverse sleep dataset, demonstrating improved generalization and performance in sleep and disease prediction tasks through optimized pre-training and scaling strategies.

Contribution

The paper provides an in-depth analysis of pre-training objectives and scaling patterns, leading to the development of OSF, a state-of-the-art sleep foundation model with enhanced generalization capabilities.

Findings

Scaling sample size, model capacity, and data sources improves performance.

Channel-invariant features are crucial for generalization.

Existing models struggle with missing channels during inference.

Abstract

Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts. There have been growing efforts to build general-purpose foundation models (FMs) for sleep physiology, but lack an in-depth understanding of the pre-training process and scaling patterns that lead to more generalizable sleep FMs. To fill this gap, we curate a massive corpus of 166,500 hours of sleep recordings from nine public sources and establish SleepBench, a comprehensive, fully open-source benchmark. Leveraging SleepBench, we systematically evaluate four families of self-supervised pre-training objectives and uncover three critical findings: (1) existing FMs fail to generalize to missing channels at inference; (2) channel-invariant feature learning is essential for pre-training; and (3) scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance.With an enhanced pre-training and scaling recipe, we introduce OSF, a family of sleep FMs that achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF also reveals intriguing properties in sample efficiency, hierarchical aggregation, and cross-dataset scaling.