Approximation-Aware Bayesian Optimization

TL;DR

This paper introduces a modified sparse variational Gaussian process approach for Bayesian optimization that aligns better with data acquisition goals, improving efficiency in high-dimensional tasks like molecular design.

Contribution

It unifies GP approximation and data acquisition into a joint optimization framework using utility-calibrated variational inference, enhancing decision-making in Bayesian optimization.

Findings

Outperforms standard SVGPs on high-dimensional benchmarks

Compatible with various acquisition functions and trust region methods

Improves data efficiency in molecular design and control tasks

Abstract

High-dimensional Bayesian optimization (BO) tasks such as molecular design often require 10,000 function evaluations before obtaining meaningful results. While methods like sparse variational Gaussian processes (SVGPs) reduce computational requirements in these settings, the underlying approximations result in suboptimal data acquisitions that slow the progress of optimization. In this paper we modify SVGPs to better align with the goals of BO: targeting informed data acquisition rather than global posterior fidelity. Using the framework of utility-calibrated variational inference, we unify GP approximation and data acquisition into a joint optimization problem, thereby ensuring optimal decisions under a limited computational budget. Our approach can be used with any decision-theoretic acquisition function and is compatible with trust region methods like TuRBO. We derive efficient joint objectives for the expected improvement and knowledge gradient acquisition functions in both the standard and batch BO settings. Our approach outperforms standard SVGPs on high-dimensional benchmark tasks in control and molecular design.