Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics

TL;DR

This study investigates whether neural scaling laws, known in NLP and vision, also apply to single-cell transcriptomics using masked-reconstruction transformers, revealing data-dependent scaling behaviors.

Contribution

First systematic analysis of scaling laws in single-cell genomics with transformers, identifying data availability as key to scaling behavior and establishing parallels with NLP models.

Findings

Scaling laws exist in data-rich regimes with a power-law relationship.

Model capacity limits are less relevant in data-scarce settings.

Estimated entropy per masked gene position is approximately 2.30 bits.

Abstract

Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored. We present the first systematic study of scaling behaviour for masked-reconstruction transformers trained on single-cell RNA sequencing (scRNA-seq) data. Using expression profiles from the CELLxGENE Census, we construct two experimental regimes: a data-rich regime (512 highly variable genes, 200,000 cells) and a data-limited regime (1,024 genes, 10,000 cells). Across seven model sizes spanning three orders of magnitude in parameter count (533 to 3.4 x 10^8 parameters), we fit the parametric scaling law to validation mean squared error (MSE). The data-rich regime exhibits clear power-law scaling with an irreducible loss floor of c ~ 1.44, while the data-limited regime shows negligible scaling, indicating that model capacity is not the binding constraint when data are scarce. These results establish that scaling laws analogous to those observed in natural language processing do emerge in single-cell transcriptomics when sufficient data are available, and they identify the data-to-parameter ratio as a critical determinant of scaling behaviour. A preliminary conversion of the data-rich asymptotic floor to information-theoretic units yields an estimate of approximately 2.30 bits of entropy per masked gene position. We discuss implications for the design of single-cell foundation models and outline the additional measurements needed to refine this entropy estimate.