Differentiating Policies for Non-Myopic Bayesian Optimization

TL;DR

This paper introduces a method to efficiently estimate and optimize non-myopic acquisition functions in Bayesian optimization, enabling better sampling policies through stochastic gradient methods.

Contribution

It presents an efficient approach to estimate rollout acquisition functions and their gradients, facilitating stochastic gradient optimization of non-myopic policies in Bayesian optimization.

Findings

Efficient estimation of rollout acquisition functions.

Gradient-based optimization of sampling policies.

Improved non-myopic Bayesian optimization performance.

Abstract

Bayesian optimization (BO) methods choose sample points by optimizing an acquisition function derived from a statistical model of the objective. These acquisition functions are chosen to balance sampling regions with predicted good objective values against exploring regions where the objective is uncertain. Standard acquisition functions are myopic, considering only the impact of the next sample, but non-myopic acquisition functions may be more effective. In principle, one could model the sampling by a Markov decision process, and optimally choose the next sample by maximizing an expected reward computed by dynamic programming; however, this is infeasibly expensive. More practical approaches, such as rollout, consider a parametric family of sampling policies. In this paper, we show how to efficiently estimate rollout acquisition functions and their gradients, enabling stochastic gradient-based optimization of sampling policies.