A Simple Approach for Non-stationary Linear Bandits

TL;DR

This paper corrects previous theoretical results on non-stationary linear bandits, proposes a simple restarted UCB algorithm with optimal regret bounds, and demonstrates its effectiveness through empirical validation.

Contribution

It identifies a flaw in existing analyses, provides a corrected regret bound, and introduces a simple restarted strategy that achieves optimal regret without complex mechanisms.

Findings

Corrected the regret analysis for non-stationary linear bandits.

Proposed a simple restarted UCB algorithm achieving $ ilde{O}(T^{3/4}P_T^{1/4})$ regret.

Empirical results confirm the effectiveness of the proposed approach.

Abstract

This paper investigates the problem of non-stationary linear bandits, where the unknown regression parameter is evolving over time. Existing studies develop various algorithms and show that they enjoy an $\widetilde{\mathcal{O}}(T^{2/3}P_T^{1/3})$ dynamic regret, where $T$ is the time horizon and $P_T$ is the path-length that measures the fluctuation of the evolving unknown parameter. In this paper, we discover that a serious technical flaw makes their results ungrounded, and then present a fix, which gives an $\widetilde{\mathcal{O}}(T^{3/4}P_T^{1/4})$ dynamic regret without modifying original algorithms. Furthermore, we demonstrate that instead of using sophisticated mechanisms, such as sliding window or weighted penalty, a simple restarted strategy is sufficient to attain the same regret guarantee. Specifically, we design an UCB-type algorithm to balance exploitation and exploration, and restart it periodically to handle the drift of unknown parameters. Our approach enjoys an $\widetilde{\mathcal{O}}(T^{3/4}P_T^{1/4})$ dynamic regret. Note that to achieve this bound, the algorithm requires an oracle knowledge of the path-length $P_T$. Combining the bandits-over-bandits mechanism by treating our algorithm as the base learner, we can further achieve the same regret bound in a parameter-free way. Empirical studies also validate the effectiveness of our approach.