Replicable Bandits

TL;DR

This paper introduces the concept of replicable policies in stochastic bandits, demonstrating that such policies can match the regret bounds of traditional policies while ensuring identical arm pulls across independent runs.

Contribution

It formally defines replicable policies in bandits and develops algorithms that achieve optimal regret bounds with high replicability, a novel combination in interactive learning.

Findings

Replicable policies exist with near-optimal regret bounds.

Algorithms achieve optimal regret with high replicability.

Replicability does not significantly compromise exploration-exploitation balance.

Abstract

In this paper, we introduce the notion of replicable policies in the context of stochastic bandits, one of the canonical problems in interactive learning. A policy in the bandit environment is called replicable if it pulls, with high probability, the exact same sequence of arms in two different and independent executions (i.e., under independent reward realizations). We show that not only do replicable policies exist, but also they achieve almost the same optimal (non-replicable) regret bounds in terms of the time horizon. More specifically, in the stochastic multi-armed bandits setting, we develop a policy with an optimal problem-dependent regret bound whose dependence on the replicability parameter is also optimal. Similarly, for stochastic linear bandits (with finitely and infinitely many arms) we develop replicable policies that achieve the best-known problem-independent regret bounds with an optimal dependency on the replicability parameter. Our results show that even though randomization is crucial for the exploration-exploitation trade-off, an optimal balance can still be achieved while pulling the exact same arms in two different rounds of executions.