On Optimality of Greedy Policy for a Class of Standard Reward Function of Restless Multi-armed Bandit Problem

TL;DR

This paper establishes conditions under which a greedy policy is optimal for a class of restless bandit problems with standard reward functions, simplifying the decision process in complex stochastic environments.

Contribution

It provides a closed-form condition on the discount factor ensuring greedy policy optimality for a specific reward class in restless bandits.

Findings

Greedy policy is optimal when the discount factor equals one.

The standard reward function allows easy assessment of greedy policy optimality.

Mathematical conditions verified with examples in cognitive radio networks.

Abstract

In this paper,we consider the restless bandit problem, which is one of the most well-studied generalizations of the celebrated stochastic multi-armed bandit problem in decision theory. However, it is known be PSPACE-Hard to approximate to any non-trivial factor. Thus the optimality is very difficult to obtain due to its high complexity. A natural method is to obtain the greedy policy considering its stability and simplicity. However, the greedy policy will result in the optimality loss for its intrinsic myopic behavior generally. In this paper, by analyzing one class of so-called standard reward function, we establish the closed-form condition about the discounted factor \beta such that the optimality of the greedy policy is guaranteed under the discounted expected reward criterion, especially, the condition \beta = 1 indicating the optimality of the greedy policy under the average accumulative reward criterion. Thus, the standard form of reward function can easily be used to judge the optimality of the greedy policy without any complicated calculation. Some examples in cognitive radio networks are presented to verify the effectiveness of the mathematical result in judging the optimality of the greedy policy.