A Simple Heuristic for Bayesian Optimization with A Low Budget

TL;DR

This paper introduces a heuristic to improve Bayesian optimization performance under low evaluation budgets by refining the search space, leading to more efficient optimization in limited-resource scenarios.

Contribution

The paper proposes a novel heuristic method that refines the search space for Bayesian optimization when evaluation budgets are low, enhancing its efficiency.

Findings

Outperforms standard Bayesian optimization in low-budget settings

Achieves comparable or better results than existing search-space division methods

Effective in hyperparameter tuning and benchmark function optimization

Abstract

The aim of black-box optimization is to optimize an objective function within the constraints of a given evaluation budget. In this problem, it is generally assumed that the computational cost for evaluating a point is large; thus, it is important to search efficiently with as low budget as possible. Bayesian optimization is an efficient method for black-box optimization and provides exploration-exploitation trade-off by constructing a surrogate model that considers uncertainty of the objective function. However, because Bayesian optimization should construct the surrogate model for the entire search space, it does not exhibit good performance when points are not sampled sufficiently. In this study, we develop a heuristic method refining the search space for Bayesian optimization when the available evaluation budget is low. The proposed method refines a promising region by dividing the original region so that Bayesian optimization can be executed with the promising region as the initial search space. We confirm that Bayesian optimization with the proposed method outperforms Bayesian optimization alone and shows equal or better performance to two search-space division algorithms through experiments on the benchmark functions and the hyperparameter optimization of machine learning algorithms.