Can Synthetic Data Improve Symbolic Regression Extrapolation Performance?

TL;DR

This paper explores whether synthetic data generated via KDE and knowledge distillation can enhance the extrapolation capabilities of symbolic regression models, especially those trained with genetic programming, across various datasets.

Contribution

It introduces a method combining KDE and teacher-student training to generate synthetic data aimed at improving GP-based symbolic regression extrapolation performance.

Findings

GP models benefit from synthetic data in extrapolation regions.

Synthetic data improves GP performance more than other models.

Extrapolation improvements depend on dataset and teacher model used.

Abstract

Many machine learning models perform well when making predictions within the training data range, but often struggle when required to extrapolate beyond it. Symbolic regression (SR) using genetic programming (GP) can generate flexible models but is prone to unreliable behaviour in extrapolation. This paper investigates whether adding synthetic data can help improve performance in such cases. We apply Kernel Density Estimation (KDE) to identify regions in the input space where the training data is sparse. Synthetic data is then generated in those regions using a knowledge distillation approach: a teacher model generates predictions on new input points, which are then used to train a student model. We evaluate this method across six benchmark datasets, using neural networks (NN), random forests (RF), and GP both as teacher models (to generate synthetic data) and as student models (trained on the augmented data). Results show that GP models can often improve when trained on synthetic data, especially in extrapolation areas. However, the improvement depends on the dataset and teacher model used. The most important improvements are observed when synthetic data from GPe is used to train GPp in extrapolation regions. Changes in interpolation areas show only slight changes. We also observe heterogeneous errors, where model performance varies across different regions of the input space. Overall, this approach offers a practical solution for better extrapolation. Note: An earlier version of this work appeared in the GECCO 2025 Workshop on Symbolic Regression. This arXiv version corrects several parts of the original submission.