Non-stationary Bandits and Meta-Learning with a Small Set of Optimal Arms

TL;DR

This paper introduces algorithms for non-stationary and meta-learning bandit problems that outperform standard methods, especially when the number of optimal arms is small relative to total arms, with regret bounds tailored to different settings.

Contribution

The paper proposes a reduction-based algorithm for non-stationary and meta-learning bandits that achieves improved regret bounds in regimes with few optimal arms.

Findings

Regret bounds are smaller than the baseline of O(\u221a{KNT}) in large task regimes.

For fixed task length, regret is bounded by O(NM77Md77) and improved to O(N77+N^{1/2}77) under additional assumptions.

Abstract

We study a sequential decision problem where the learner faces a sequence of $K$-armed bandit tasks. The task boundaries might be known (the bandit meta-learning setting), or unknown (the non-stationary bandit setting). For a given integer $M\le K$, the learner aims to compete with the best subset of arms of size $M$. We design an algorithm based on a reduction to bandit submodular maximization, and show that, for $T$ rounds comprised of $N$ tasks, in the regime of large number of tasks and small number of optimal arms $M$, its regret in both settings is smaller than the simple baseline of $\tilde{O}(\sqrt{KNT})$ that can be obtained by using standard algorithms designed for non-stationary bandit problems. For the bandit meta-learning problem with fixed task length $\tau$, we show that the regret of the algorithm is bounded as $\tilde{O}(NM\sqrt{M \tau}+N^{2/3}M\tau)$. Under additional assumptions on the identifiability of the optimal arms in each task, we show a bandit meta-learning algorithm with an improved $\tilde{O}(N\sqrt{M \tau}+N^{1/2}\sqrt{M K \tau})$ regret.