Non-Stationary Dueling Bandits

TL;DR

This paper addresses the non-stationary dueling bandits problem by proposing algorithms that adapt to changing preference matrices, providing regret bounds and a lower bound, improving upon existing methods in dynamic environments.

Contribution

It introduces the Beat the Winner Reset algorithm with tighter regret bounds and two meta-algorithms, DETECT and Monitored Dueling Bandits, for handling non-stationarity without prior knowledge of segment count or length.

Findings

Beat the Winner Reset achieves tighter regret bounds in stationary cases.

Meta-algorithms effectively detect changes and adapt without prior knowledge.

Theoretical lower bounds establish limits for non-stationary dueling bandits.

Abstract

We study the non-stationary dueling bandits problem with $K$ arms, where the time horizon $T$ consists of $M$ stationary segments, each of which is associated with its own preference matrix. The learner repeatedly selects a pair of arms and observes a binary preference between them as feedback. To minimize the accumulated regret, the learner needs to pick the Condorcet winner of each stationary segment as often as possible, despite preference matrices and segment lengths being unknown. We propose the $\mathrm{Beat\, the\, Winner\, Reset}$ algorithm and prove a bound on its expected binary weak regret in the stationary case, which tightens the bound of current state-of-art algorithms. We also show a regret bound for the non-stationary case, without requiring knowledge of $M$ or $T$. We further propose and analyze two meta-algorithms, $\mathrm{DETECT}$ for weak regret and $\mathrm{Monitored\, Dueling\, Bandits}$ for strong regret, both based on a detection-window approach that can incorporate any dueling bandit algorithm as a black-box algorithm. Finally, we prove a worst-case lower bound for expected weak regret in the non-stationary case.