On the Sublinear Regret of GP-UCB

TL;DR

This paper proves that the GP-UCB algorithm achieves nearly optimal sublinear regret in kernelized bandit problems, including for commonly used kernels like Matérn, by introducing a new analysis technique based on regularizing kernel ridge estimators.

Contribution

The paper provides the first tight regret bounds for GP-UCB, improving analysis for Matérn kernels and resolving a longstanding open problem in the field.

Findings

GP-UCB achieves nearly optimal sublinear regret.

Improved regret bounds for Matérn kernels.

New analysis technique using regularized kernel ridge estimators.

Abstract

In the kernelized bandit problem, a learner aims to sequentially compute the optimum of a function lying in a reproducing kernel Hilbert space given only noisy evaluations at sequentially chosen points. In particular, the learner aims to minimize regret, which is a measure of the suboptimality of the choices made. Arguably the most popular algorithm is the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm, which involves acting based on a simple linear estimator of the unknown function. Despite its popularity, existing analyses of GP-UCB give a suboptimal regret rate, which fails to be sublinear for many commonly used kernels such as the Mat\'ern kernel. This has led to a longstanding open question: are existing regret analyses for GP-UCB tight, or can bounds be improved by using more sophisticated analytical techniques? In this work, we resolve this open question and show that GP-UCB enjoys nearly optimal regret. In particular, our results yield sublinear regret rates for the Mat\'ern kernel, improving over the state-of-the-art analyses and partially resolving a COLT open problem posed by Vakili et al. Our improvements rely on a key technical contribution -- regularizing kernel ridge estimators in proportion to the smoothness of the underlying kernel $k$. Applying this key idea together with a largely overlooked concentration result in separable Hilbert spaces (for which we provide an independent, simplified derivation), we are able to provide a tighter analysis of the GP-UCB algorithm.