Optimality of Myopic Policy for Restless Multiarmed Bandit with Imperfect Observation

TL;DR

This paper analyzes a restless multi-armed bandit problem with imperfect observations, providing conditions under which a simple myopic policy is proven to be optimal, thus offering a computationally feasible solution.

Contribution

The paper derives analytical, closed-form conditions that guarantee the optimality of the myopic policy in a complex RMAB setting with imperfect observations.

Findings

Conditions for myopic policy optimality derived

Analytical solutions for a class of RMAB problems

Simplified policy implementation with proven optimality

Abstract

We consider the scheduling problem concerning N projects. Each project evolves as a multi-state Markov process. At each time instant, one project is scheduled to work, and some reward depending on the state of the chosen project is obtained. The objective is to design a scheduling policy that maximizes the expected accumulated discounted reward over a finite or infinite horizon. The considered problem can be cast into a restless multi-armed bandit (RMAB) problem that is of fundamental importance in decision theory. It is well-known that solving the RMAB problem is PSPACE-hard, with the optimal policy usually intractable due to the exponential computation complexity. A natural alternative is to consider the easily implementable myopic policy that maximizes the immediate reward. In this paper, we perform an analytical study on the considered RMAB problem, and establish a set of closed-form conditions to guarantee the optimality of the myopic policy.