Bayesian Algorithms for Adversarial Online Learning: from Finite to Infinite Action Spaces

TL;DR

This paper introduces a Bayesian Thompson sampling approach for adversarial online learning, extending to infinite action spaces, and demonstrates near-optimal regret bounds in high-dimensional continuous settings.

Contribution

It develops a novel Thompson sampling framework that models the adversary's actions directly, achieving regret bounds in infinite action spaces using Gaussian process priors.

Findings

Recovers optimal rates in finite expert settings.

Provides regret bounds for continuous action spaces with Gaussian process priors.

Extends Bayesian online learning to uncountably infinite action spaces.

Abstract

We develop a form Thompson sampling for online learning under full feedback - also known as prediction with expert advice - where the learner's prior is defined over the space of an adversary's future actions, rather than the space of experts. We show regret decomposes into regret the learner expected a priori, plus a prior-robustness-type term we call excess regret. In the classical finite-expert setting, this recovers optimal rates. As an initial step towards practical online learning in settings with a potentially-uncountably-infinite number of experts, we show that Thompson sampling over the $d$-dimensional unit cube, using a certain Gaussian process prior widely-used in the Bayesian optimization literature, has a $\mathcal{O}\Big(\beta\sqrt{Td\log(1+\sqrt{d}\frac{\lambda}{\beta})}\Big)$ rate against a $\beta$-bounded $\lambda$-Lipschitz adversary.