Optimal Regret for Policy Optimization in Contextual Bandits

TL;DR

This paper establishes the first high-probability optimal regret bound for policy optimization in stochastic contextual bandits with function approximation, bridging theory and practice.

Contribution

It provides the first rigorous optimal regret analysis for policy optimization methods in contextual bandits with general offline function approximation.

Findings

Achieves an optimal regret bound of () () log ||)

Algorithm is both efficient and theoretically optimal

Empirical evaluation supports theoretical claims

Abstract

We present the first high-probability optimal regret bound for a policy optimization technique applied to the problem of stochastic contextual multi-armed bandit (CMAB) with general offline function approximation. Our algorithm is both efficient and achieves an optimal regret bound of $\widetilde{O}(\sqrt{ K|\mathcal{A}|\log|\mathcal{F}|})$, where $K$ is the number of rounds, $\mathcal{A}$ is the set of arms, and $\mathcal{F}$ is the function class used to approximate the losses. Our results bridge the gap between theory and practice, demonstrating that the widely used policy optimization methods for the contextual bandit problem can achieve a rigorously-proved optimal regret bound. We support our theoretical results with an empirical evaluation of our algorithm.