Nonparametric Bayesian Optimization for General Rewards

TL;DR

This paper introduces a novel nonparametric Bayesian optimization algorithm using an infinite Gaussian process, providing no-regret guarantees for general reward functions and demonstrating superior empirical performance in complex reward scenarios.

Contribution

It presents the first BO algorithm with no-regret guarantees for broad reward models using a new infinite Gaussian process surrogate, extending Bayesian optimization capabilities.

Findings

Achieves no-regret guarantee in general reward settings

Handles non-stationary and heavy-tailed rewards effectively

Demonstrates state-of-the-art empirical performance

Abstract

This work focuses on Bayesian optimization (BO) under reward model uncertainty. We propose the first BO algorithm that achieves no-regret guarantee in a general reward setting, requiring only Lipschitz continuity of the objective function and accommodating a broad class of measurement noise. The core of our approach is a novel surrogate model, termed as infinite Gaussian process ($\infty$-GP). It is a Bayesian nonparametric model that places a prior on the space of reward distributions, enabling it to represent a substantially broader class of reward models than classical Gaussian process (GP). The $\infty$-GP is used in combination with Thompson Sampling (TS) to enable effective exploration and exploitation. Correspondingly, we develop a new TS regret analysis framework for general rewards, which relates the regret to the total variation distance between the surrogate model and the true reward distribution. Furthermore, with a truncated Gibbs sampling procedure, our method is computationally scalable, incurring minimal additional memory and computational complexities compared to classical GP. Empirical results demonstrate state-of-the-art performance, particularly in settings with non-stationary, heavy-tailed, or other ill-conditioned rewards.