Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization

TL;DR

This paper introduces a novel method for Bayesian optimization that decouples optimizer updates from acquisition function evaluations, significantly speeding up the process while maintaining convergence guarantees.

Contribution

It proposes decoupling quasi-Newton updates from acquisition function evaluations, reducing computational overhead in Bayesian optimization.

Findings

Decoupling optimizer updates speeds up Bayesian optimization.

The method maintains theoretical convergence guarantees.

Implementation in GPSampler reduces overall computational time.

Abstract

Bayesian optimization (BO) efficiently finds high-performing parameters by maximizing an acquisition function, which models the promise of parameters. A major computational bottleneck arises in acquisition function optimization, where multi-start optimization (MSO) with quasi-Newton (QN) methods is required due to the non-convexity of the acquisition function. BoTorch, a widely used BO library, currently optimizes the summed acquisition function over multiple points, leading to the speedup of MSO owing to PyTorch batching. Nevertheless, this paper empirically demonstrates the suboptimality of this approach in terms of off-diagonal approximation errors in the inverse Hessian of a QN method, slowing down its convergence. To address this problem, we propose to decouple QN updates using a coroutine while batching the acquisition function calls. Our approach not only yields the theoretically identical convergence to the sequential MSO but also drastically reduces the wall-clock time compared to the previous approaches. Our approach is available in GPSampler in Optuna, effectively reducing its computational overhead.