Regret Bounds for Restless Markov Bandits

TL;DR

This paper introduces an algorithm for the restless Markov bandit problem that achieves near-optimal regret bounds without strong assumptions on the Markov chains, highlighting the suboptimality of index-based policies.

Contribution

The paper presents a new algorithm with $ ilde{O}( oot{T})$ regret for restless Markov bandits, and proves index policies are suboptimal in this setting.

Findings

Achieves $ ilde{O}( oot{T})$ regret bound.

Index policies are suboptimal for restless Markov bandits.

No assumptions other than irreducibility on Markov chains.

Abstract

We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after $T$ steps achieves $\tilde{O}(\sqrt{T})$ regret with respect to the best policy that knows the distributions of all arms. No assumptions on the Markov chains are made except that they are irreducible. In addition, we show that index-based policies are necessarily suboptimal for the considered problem.