Nonstationary Stochastic Multiarmed Bandits: UCB Policies and Minimax Regret

TL;DR

This paper introduces and analyzes UCB-based algorithms for nonstationary stochastic multi-armed bandit problems, achieving order-optimal minimax regret under variation constraints and handling heavy-tailed rewards.

Contribution

It extends UCB policies with resetting, sliding windows, and discounting to nonstationary environments and develops robust versions for heavy-tailed rewards.

Findings

Proposed policies are order-optimal in worst-case regret.

Algorithms effectively adapt to reward distribution changes.

Robust methods handle heavy-tailed reward distributions.

Abstract

We study the nonstationary stochastic Multi-Armed Bandit (MAB) problem in which the distribution of rewards associated with each arm are assumed to be time-varying and the total variation in the expected rewards is subject to a variation budget. The regret of a policy is defined by the difference in the expected cumulative rewards obtained using the policy and using an oracle that selects the arm with the maximum mean reward at each time. We characterize the performance of the proposed policies in terms of the worst-case regret, which is the supremum of the regret over the set of reward distribution sequences satisfying the variation budget. We extend Upper-Confidence Bound (UCB)-based policies with three different approaches, namely, periodic resetting, sliding observation window and discount factor and show that they are order-optimal with respect to the minimax regret, i.e., the minimum worst-case regret achieved by any policy. We also relax the sub-Gaussian assumption on reward distributions and develop robust versions the proposed polices that can handle heavy-tailed reward distributions and maintain their performance guarantees.