Sequential Decision Making with Limited Observation Capability: Application to Wireless Networks

TL;DR

This paper introduces lazy restless bandits, a new class of decision-making models with hidden states and cumulative feedback, and analyzes optimal and index-based policies for maximizing long-term rewards in wireless network applications.

Contribution

It defines lazy restless bandits with cumulative feedback, proves the existence of threshold policies, and develops index policies with closed-form expressions and algorithms for complex cases.

Findings

Optimal policies have threshold structure in belief space.

Whittle-index policy is shown to be indexable with closed-form expressions for special cases.

Simulation results compare the effectiveness of different policies and provide bounds on optimality.

Abstract

This work studies a generalized class of restless multi-armed bandits with hidden states and allow cumulative feedback, as opposed to the conventional instantaneous feedback. We call them lazy restless bandits (LRB) as the events of decision-making are sparser than events of state transition. Hence, feedback after each decision event is the cumulative effect of the following state transition events. The states of arms are hidden from the decision-maker and rewards for actions are state dependent. The decision-maker needs to choose one arm in each decision interval, such that long term cumulative reward is maximized. As the states are hidden, the decision-maker maintains and updates its belief about them. It is shown that LRBs admit an optimal policy which has threshold structure in belief space. The Whittle-index policy for solving LRB problem is analyzed; indexability of LRBs is shown. Further, closed-form index expressions are provided for two sets of special cases; for more general cases, an algorithm for index computation is provided. An extensive simulation study is presented; Whittle-index, modified Whittle-index and myopic policies are compared. Lagrangian relaxation of the problem provides an upper bound on the optimal value function; it is used to assess the degree of sub-optimality various policies.