Data-Efficient Symbolic Regression via Foundation Model Distillation

TL;DR

EQUATE is a novel framework that fine-tunes foundation models for symbolic regression in low-data scenarios by combining symbolic-numeric alignment with embedding optimization, leading to improved accuracy and robustness.

Contribution

We introduce EQUATE, a data-efficient fine-tuning method that reformulates symbolic search as a continuous optimization in embedding space, enhancing symbolic regression from limited data.

Findings

EQUATE outperforms state-of-the-art baselines in accuracy and robustness.

EQUATE maintains low complexity and enables fast inference.

EQUATE demonstrates effectiveness across multiple benchmark datasets.

Abstract

Discovering interpretable mathematical equations from observed data (a.k.a. equation discovery or symbolic regression) is a cornerstone of scientific discovery, enabling transparent modeling of physical, biological, and economic systems. While foundation models pre-trained on large-scale equation datasets offer a promising starting point, they often suffer from negative transfer and poor generalization when applied to small, domain-specific datasets. In this paper, we introduce EQUATE (Equation Generation via QUality-Aligned Transfer Embeddings), a data-efficient fine-tuning framework that adapts foundation models for symbolic equation discovery in low-data regimes via distillation. EQUATE combines symbolic-numeric alignment with evaluator-guided embedding optimization, enabling a principled embedding-search-generation paradigm. Our approach reformulates discrete equation search as a continuous optimization task in a shared embedding space, guided by data-equation fitness and simplicity. Experiments across three standard public benchmarks (Feynman, Strogatz, and black-box datasets) demonstrate that EQUATE consistently outperforms state-of-the-art baselines in both accuracy and robustness, while preserving low complexity and fast inference. These results highlight EQUATE as a practical and generalizable solution for data-efficient symbolic regression in foundation model distillation settings.