Restless Linear Bandits

TL;DR

This paper extends linear bandit models to account for time-dependent parameters with dependencies, proposing an algorithm with sub-linear regret that handles long-range dependencies using coupling techniques.

Contribution

It introduces a generalized restless linear bandit framework with dependencies, and proposes LinMix-UCB, an algorithm with provable regret bounds under exponential mixing conditions.

Findings

Proposed a new restless linear bandit model with dependent parameters.

Developed LinMix-UCB algorithm with sub-linear regret guarantees.

Demonstrated robustness against long-range dependencies using coupling methods.

Abstract

A more general formulation of the linear bandit problem is considered to allow for dependencies over time. Specifically, it is assumed that there exists an unknown $\mathbb{R}^d$-valued stationary $\varphi$-mixing sequence of parameters $(\theta_t,~t \in \mathbb{N})$ which gives rise to pay-offs. This instance of the problem can be viewed as a generalization of both the classical linear bandits with iid noise, and the finite-armed restless bandits. In light of the well-known computational hardness of optimal policies for restless bandits, an approximation is proposed whose error is shown to be controlled by the $\varphi$-dependence between consecutive $\theta_t$. An optimistic algorithm, called LinMix-UCB, is proposed for the case where $\theta_t$ has an exponential mixing rate. The proposed algorithm is shown to incur a sub-linear regret of $\mathcal{O}\left(\sqrt{d n\mathrm{polylog}(n) }\right)$ with respect to an oracle that always plays a multiple of $\mathbb{E}\theta_t$. The main challenge in this setting is to ensure that the exploration-exploitation strategy is robust against long-range dependencies. The proposed method relies on Berbee's coupling lemma to carefully select near-independent samples and construct confidence ellipsoids around empirical estimates of $\mathbb{E}\theta_t$.