A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning

TL;DR

This paper introduces a novel hybrid approach combining linear programming with genetic algorithms to improve hyperparameter tuning efficiency in machine learning models, applicable to both training and fine-tuning stages.

Contribution

It formulates a linear programming method that enhances genetic algorithms for faster, more precise hyperparameter search, adaptable to various models and search techniques.

Findings

Significant improvement in hyperparameter search speed.

Effective fine-tuning on trained models.

Demonstrated success on MNIST and CIFAR-10 datasets.

Abstract

In this paper, we formulate the hyperparameter tuning problem in machine learning as a bilevel program. The bilevel program is solved using a micro genetic algorithm that is enhanced with a linear program. While the genetic algorithm searches over discrete hyperparameters, the linear program enhancement allows hyper local search over continuous hyperparameters. The major contribution in this paper is the formulation of a linear program that supports fast search over continuous hyperparameters, and can be integrated with any hyperparameter search technique. It can also be applied directly on any trained machine learning or deep learning model for the purpose of fine-tuning. We test the performance of the proposed approach on two datasets, MNIST and CIFAR-10. Our results clearly demonstrate that using the linear program enhancement offers significant promise when incorporated with any population-based approach for hyperparameter tuning.