Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Bandit

TL;DR

This paper introduces M-UCB, a change-detection integrated algorithm for piecewise-stationary bandits, achieving near-optimal regret bounds and demonstrating superior empirical performance over existing methods.

Contribution

The paper proposes a simple yet effective change-detection method integrated with UCB, achieving nearly optimal regret bounds for non-stationary bandit problems.

Findings

M-UCB achieves regret of order $O( oot{MKT}\log T)$, nearly matching the lower bound.

M-UCB outperforms state-of-the-art algorithms in numerical experiments.

The method effectively detects and adapts to changes in reward distributions.

Abstract

Multi-armed bandit (MAB) is a class of online learning problems where a learning agent aims to maximize its expected cumulative reward while repeatedly selecting to pull arms with unknown reward distributions. We consider a scenario where the reward distributions may change in a piecewise-stationary fashion at unknown time steps. We show that by incorporating a simple change-detection component with classic UCB algorithms to detect and adapt to changes, our so-called M-UCB algorithm can achieve nearly optimal regret bound on the order of $O(\sqrt{MKT\log T})$, where $T$ is the number of time steps, $K$ is the number of arms, and $M$ is the number of stationary segments. Comparison with the best available lower bound shows that our M-UCB is nearly optimal in $T$ up to a logarithmic factor. We also compare M-UCB with the state-of-the-art algorithms in numerical experiments using a public Yahoo! dataset to demonstrate its superior performance.