Searching in the Forest for Local Bayesian Optimization

TL;DR

This paper introduces BOinG, a two-stage Bayesian optimization method that leverages the benign landscape of hyperparameter optimization problems, using a global random forest model to identify promising regions and a local model for efficient search.

Contribution

The paper presents BOinG, a novel two-stage BO approach tailored for mid-sized HPO problems, combining global and local models to improve efficiency.

Findings

BOinG outperforms baseline methods on synthetic functions.

BOinG is effective on mid-sized HPO problems.

The approach exploits the structure of typical HPO landscapes.

Abstract

Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments showed that the loss landscapes of HPO problems tend to be more benign than previously assumed, i.e. in the best case uni-modal and convex, such that a BO framework could be more efficient if it can focus on those promising local regions. In this paper, we propose BOinG, a two-stage approach that is tailored toward mid-sized configuration spaces, as one encounters in many HPO problems. In the first stage, we build a scalable global surrogate model with a random forest to describe the overall landscape structure. Further, we choose a promising subregion via a bottom-up approach on the upper-level tree structure. In the second stage, a local model in this subregion is utilized to suggest the point to be evaluated next. Empirical experiments show that BOinG is able to exploit the structure of typical HPO problems and performs particularly well on mid-sized problems from synthetic functions and HPO.