Neural Symbolic Regression that Scales

TL;DR

This paper presents a scalable neural symbolic regression method that uses large-scale pre-training of a Transformer model to discover underlying equations from data, improving with more data and computation.

Contribution

It introduces the first pre-trained Transformer approach for symbolic regression, capable of discovering physical equations and scaling with data and compute.

Findings

Successfully rediscovered well-known physical equations

Outperformed traditional symbolic regression methods

Improved accuracy with increased data and compute

Abstract

Symbolic equations are at the core of scientific discovery. The task of discovering the underlying equation from a set of input-output pairs is called symbolic regression. Traditionally, symbolic regression methods use hand-designed strategies that do not improve with experience. In this paper, we introduce the first symbolic regression method that leverages large scale pre-training. We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs. At test time, we query the model on a new set of points and use its output to guide the search for the equation. We show empirically that this approach can re-discover a set of well-known physical equations, and that it improves over time with more data and compute.