Genetic Algorithms in Regression

TL;DR

This paper introduces GAReg, a genetic algorithm package designed for discrete optimization in regression problems, effectively handling complex, high-dimensional search spaces where traditional methods struggle.

Contribution

GAReg provides a unified, flexible genetic algorithm framework for discrete regression optimization, supporting various model selection and knot placement tasks with efficient search capabilities.

Findings

GAReg finds near-optimal solutions in high-dimensional spaces.

The package supports multiple genetic operators and parallelization.

It outperforms exhaustive and traditional methods in complex regression tasks.

Abstract

Many statistical problems involve optimization over a discrete parameter space having an unknown dimension. In such settings, gradient-based methods often fail due to the non-differentiability of the objective function or a non-convex or massive search space with an objective function having many local maxima/minima. This paper presents GAReg, a unified genetic algorithm package that handles discrete optimization regression problems, which works well when standard algorithms are unjustified. GAReg provides a compact chromosome representation supporting optimal knot placement for regression splines, best-subset regression variable selection, and related problems. The package allows for uniform initialization, constraint-preserving crossover and mutation, steady-state replacement, and an optional island-model parallelization. GAReg efficiently searches high-dimensional model spaces, providing near-optimal solutions in settings where exhaustive enumeration or integer or dynamic programming approaches are infeasible.