A PDE-Based Analysis of the Symmetric Two-Armed Bernoulli Bandit

TL;DR

This paper analyzes the symmetric two-armed Bernoulli bandit problem using PDE methods, deriving explicit regret bounds and solutions to a heat equation in various regimes, advancing understanding of regret behavior in complex bandit settings.

Contribution

It introduces a PDE-based framework to explicitly compute leading order regret terms in symmetric Bernoulli bandits across different regimes, improving upon prior asymptotic results.

Findings

Explicit leading order regret terms derived for three regimes.

New non-asymptotic bounds for finite time horizons.

Connection of regret analysis to solutions of a linear heat equation.

Abstract

This work addresses a version of the two-armed Bernoulli bandit problem where the sum of the means of the arms is one (the symmetric two-armed Bernoulli bandit). In a regime where the gap between these means goes to zero as the number of prediction periods approaches infinity, i.e., the difficulty of detecting the gap increases as the sample size increases, we obtain the leading order terms of the minmax optimal regret and pseudoregret for this problem by associating each of them with a solution of a linear heat equation. Our results improve upon the previously known results; specifically, we explicitly compute these leading order terms in three different scaling regimes for the gap. Additionally, we obtain new non-asymptotic bounds for any given time horizon. Although optimal player strategies are not known for more general bandit problems, there is significant interest in considering how regret accumulates under specific player strategies, even when they are not known to be optimal. We expect that the methods of this paper should be useful in settings of that type.