Unleashing the Potential of Acquisition Functions in High-Dimensional Bayesian Optimization

TL;DR

This paper addresses the challenge of maximizing acquisition functions in high-dimensional Bayesian optimization by proposing a heuristic initialization method, which significantly improves optimization performance over traditional random strategies.

Contribution

It introduces a novel initialization approach using multiple heuristics to better exploit acquisition functions in high-dimensional BO, outperforming existing methods.

Findings

Heuristic initialization improves AF maximization in high dimensions.

The proposed method outperforms state-of-the-art approaches on synthetic and real-world tasks.

Random initialization often fails to fully utilize AF potential.

Abstract

Bayesian optimization (BO) is widely used to optimize expensive-to-evaluate black-box functions.BO first builds a surrogate model to represent the objective function and assesses its uncertainty. It then decides where to sample by maximizing an acquisition function (AF) based on the surrogate model. However, when dealing with high-dimensional problems, finding the global maximum of the AF becomes increasingly challenging. In such cases, the initialization of the AF maximizer plays a pivotal role, as an inadequate setup can severely hinder the effectiveness of the AF. This paper investigates a largely understudied problem concerning the impact of AF maximizer initialization on exploiting AFs' capability. Our large-scale empirical study shows that the widely used random initialization strategy often fails to harness the potential of an AF. In light of this, we propose a better initialization approach by employing multiple heuristic optimizers to leverage the historical data of black-box optimization to generate initial points for the AF maximize. We evaluate our approach with a range of heavily studied synthetic functions and real-world applications. Experimental results show that our techniques, while simple, can significantly enhance the standard BO and outperform state-of-the-art methods by a large margin in most test cases.