Hyper-parameter estimation method with particle swarm optimization

TL;DR

This paper introduces a novel hyper-parameter estimation method that combines particle swarm optimization with Bayesian optimization to improve performance in classification and regression tasks.

Contribution

It proposes using PSO to optimize the acquisition function within the Bayesian optimization framework for better hyper-parameter tuning.

Findings

Improved performance on benchmark classification problems

Enhanced results in regression models

Demonstrated effectiveness of PSO in hyper-parameter optimization

Abstract

Particle swarm optimization (PSO) method cannot be directly used in the problem of hyper-parameter estimation since the mathematical formulation of the mapping from hyper-parameters to loss function or generalization accuracy is unclear. Bayesian optimization (BO) framework is capable of converting the optimization of the hyper-parameters into the optimization of an acquisition function. The acquisition function is non-convex and multi-peak. So the problem can be better solved by the PSO. The proposed method in this paper uses the particle swarm method to optimize the acquisition function in the BO framework to get better hyper-parameters. The performances of proposed method in both of the classification and regression models are evaluated and demonstrated. The results on several benchmark problems are improved.