Adaptive Optimizer for Automated Hyperparameter Optimization Problem

TL;DR

This paper introduces a general framework for adaptive optimizers that automatically select algorithms and parameters during hyperparameter optimization, enhancing efficiency and parallelization capabilities.

Contribution

The paper proposes a novel adaptive optimizer framework that dynamically adjusts algorithms and parameters, demonstrated through a genetic algorithm-based example integrated with Bayesian Optimization.

Findings

Effective in parallel hyperparameter optimization

Outperforms original optimizers in experiments

Flexible framework adaptable to various optimization methods

Abstract

The choices of hyperparameters have critical effects on the performance of machine learning models. In this paper, we present a general framework that is able to construct an adaptive optimizer, which automatically adjust the appropriate algorithm and parameters in the process of optimization. Examining the method of adaptive optimizer, we product an example of using genetic algorithm to construct an adaptive optimizer based on Bayesian Optimizer and compared effectiveness with original optimizer. Especially, It has great advantages in parallel optimization.