Restless dependent bandits with fading memory

TL;DR

This paper extends multi-armed bandit algorithms to dependent data generated by weak mixing processes, showing that regret bounds are similar to the i.i.d. case under certain mixing conditions.

Contribution

It introduces a -Mix Improved UCB algorithm and analyzes its regret in dependent settings, revealing surprising results in slow-mixing scenarios.

Findings

Regret bounds similar to i.i.d. case in fast-mixing scenarios

Additive regret term in slow-mixing scenarios independent of number of arms

Lower bounds matching upper bounds up to a  log(T) factor

Abstract

We study the stochastic multi-armed bandit problem in the case when the arm samples are dependent over time and generated from so-called weak $\cC$-mixing processes. We establish a $\cC-$Mix Improved UCB agorithm and provide both problem-dependent and independent regret analysis in two different scenarios. In the first, so-called fast-mixing scenario, we show that pseudo-regret enjoys the same upper bound (up to a factor) as for i.i.d. observations; whereas in the second, slow mixing scenario, we discover a surprising effect, that the regret upper bound is similar to the independent case, with an incremental {\em additive} term which does not depend on the number of arms. The analysis of slow mixing scenario is supported with a minmax lower bound, which (up to a $\log(T)$ factor) matches the obtained upper bound.