Optimal Adaptive Learning in Uncontrolled Restless Bandit Problems

TL;DR

This paper introduces a learning algorithm for uncontrolled restless bandit problems that achieves logarithmic regret, extending adaptive learning techniques from MDPs to POMDPs in a setting with unknown transition probabilities.

Contribution

It proposes a novel algorithm for uncontrolled restless bandits with proven logarithmic regret, advancing adaptive learning in POMDP-like environments.

Findings

Achieves logarithmic regret uniformly over time.

Extends adaptive learning from MDPs to POMDPs.

Provides theoretical guarantees for unknown transition probabilities.

Abstract

In this paper we consider the problem of learning the optimal policy for uncontrolled restless bandit problems. In an uncontrolled restless bandit problem, there is a finite set of arms, each of which when pulled yields a positive reward. There is a player who sequentially selects one of the arms at each time step. The goal of the player is to maximize its undiscounted reward over a time horizon T. The reward process of each arm is a finite state Markov chain, whose transition probabilities are unknown by the player. State transitions of each arm is independent of the selection of the player. We propose a learning algorithm with logarithmic regret uniformly over time with respect to the optimal finite horizon policy. Our results extend the optimal adaptive learning of MDPs to POMDPs.