Sliding-Window Thompson Sampling for Non-Stationary Settings

TL;DR

This paper introduces a new analysis of sliding-window Thompson sampling algorithms for non-stationary multi-armed bandits, providing unified regret bounds applicable to various environments and demonstrating competitive performance in simulations.

Contribution

It offers the first unified regret analysis for sliding-window Thompson sampling algorithms in non-stationary settings, generalizing previous results and introducing complexity indices.

Findings

Unified regret bounds for NS-MABs with Bernoulli or subgaussian rewards.

Matching state-of-the-art results in abrupt and smooth change environments.

Competitive performance of proposed algorithms in simulations.

Abstract

Non-stationary multi-armed bandits (NS-MABs) model sequential decision-making problems in which the expected rewards of a set of actions, a.k.a.~arms, evolve over time. In this paper, we fill a gap in the literature by providing a novel analysis of Thompson sampling-inspired (TS) algorithms for NS-MABs that both corrects and generalizes existing work. Specifically, we study the cumulative frequentist regret of two algorithms based on sliding-window TS approaches with different priors, namely $\textit{Beta-SWTS}$ and $\textit{$\gamma$-SWGTS}$. We derive a unifying regret upper bound for these algorithms that applies to any arbitrary NS-MAB (with either Bernoulli or subgaussian rewards). Our result introduces new indices that capture the inherent sources of complexity in the learning problem. Then, we specialize our general result to two of the most common NS-MAB settings: the $\textit{abruptly changing}$ and the $\textit{smoothly changing}$ environments, showing that it matches state-of-the-art results. Finally, we evaluate the performance of the analyzed algorithms in simulated environments and compare them with state-of-the-art approaches for NS-MABs.