On the Generalization Bounds of Symbolic Regression with Genetic Programming

TL;DR

This paper provides a theoretical analysis of symbolic regression with genetic programming, deriving generalization bounds that explain how structural constraints and stability mechanisms influence model performance.

Contribution

It introduces a learning-theoretic generalization bound for GP-based SR, linking practical design choices to complexity measures and explaining empirical behaviors.

Findings

Structural restrictions reduce hypothesis class complexity.

Stability mechanisms control prediction sensitivity.

Theoretical bounds explain practices like parsimony pressure and depth limits.

Abstract

Symbolic regression (SR) with genetic programming (GP) aims to discover interpretable mathematical expressions directly from data. Despite its strong empirical success, the theoretical understanding of why GP-based SR generalizes beyond the training data remains limited. In this work, we provide a learning-theoretic analysis of SR models represented as expression trees. We derive a generalization bound for GP-style SR under constraints on tree size, depth, and learnable constants. Our result decomposes the generalization gap into two interpretable components: a structure-selection term, reflecting the combinatorial complexity of choosing an expression-tree structure, and a constant-fitting term, capturing the complexity of optimizing numerical constants within a fixed structure. This decomposition provides a theoretical perspective on several widely used practices in GP, including parsimony pressure, depth limits, numerically stable operators, and interval arithmetic. In particular, our analysis shows how structural restrictions reduce hypothesis-class growth while stability mechanisms control the sensitivity of predictions to parameter perturbations. By linking these practical design choices to explicit complexity terms in the generalization bound, our work offers a principled explanation for commonly observed empirical behaviors in GP-based SR and contributes towards a more rigorous understanding of its generalization properties.