A Diffusion Analysis of Policy Gradient for Stochastic Bandits

TL;DR

This paper analyzes the behavior of policy gradient algorithms in stochastic bandits using a continuous-time diffusion approximation, providing regret bounds and demonstrating limitations for certain learning rates.

Contribution

It introduces a diffusion approximation framework for policy gradient in stochastic bandits and derives regret bounds, highlighting the impact of learning rate choices.

Findings

Regret is bounded by O(k log(k) log(n) / η) with an appropriate learning rate η.

A constructed instance shows linear regret unless η = O(Δ^2).

The analysis reveals limitations of policy gradient methods under certain parameter settings.

Abstract

We study a continuous-time diffusion approximation of policy gradient for $k$-armed stochastic bandits. We prove that with a learning rate $\eta = O(\Delta^2/\log(n))$ the regret is $O(k \log(k) \log(n) / \eta)$ where $n$ is the horizon and $\Delta$ the minimum gap. Moreover, we construct an instance with only logarithmically many arms for which the regret is linear unless $\eta = O(\Delta^2)$.