Local Optimization Often is Ill-conditioned in Genetic Programming for Symbolic Regression

TL;DR

This paper investigates the frequent occurrence of ill-conditioned Jacobian matrices in gradient-based local optimization within genetic programming for symbolic regression, highlighting the importance of problem scaling and conditioning.

Contribution

It introduces a method using singular value decomposition to analyze Jacobian matrices and demonstrates the prevalence of ill-conditioning in practical GP applications.

Findings

Rank-deficient Jacobians are common across datasets.

Ill-conditioning decreases with smaller GP trees.

Using diverse nonlinear functions improves conditioning.

Abstract

Gradient-based local optimization has been shown to improve results of genetic programming (GP) for symbolic regression. Several state-of-the-art GP implementations use iterative nonlinear least squares (NLS) algorithms such as the Levenberg-Marquardt algorithm for local optimization. The effectiveness of NLS algorithms depends on appropriate scaling and conditioning of the optimization problem. This has so far been ignored in symbolic regression and GP literature. In this study we use a singular value decomposition of NLS Jacobian matrices to determine the numeric rank and the condition number. We perform experiments with a GP implementation and six different benchmark datasets. Our results show that rank-deficient and ill-conditioned Jacobian matrices occur frequently and for all datasets. The issue is less extreme when restricting GP tree size and when using many non-linear functions in the function set.