Towards Scaling Laws for Symbolic Regression

TL;DR

This paper investigates how the performance of deep learning-based symbolic regression scales with compute, revealing power-law relationships and optimal hyperparameters, thus providing a foundation for future model development.

Contribution

It is the first systematic study of scaling laws in symbolic regression using transformer models, identifying how compute influences performance and hyperparameter optimization.

Findings

Validation loss and success rate follow power-law trends with compute.

Optimal batch size and learning rate increase with model size.

A token-to-parameter ratio of approximately 15 is optimal.

Abstract

Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of $\approx$15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models.