Partially Observable Restless Bandits for Age-Optimal Scheduling over Markov Channels

TL;DR

This paper addresses age-optimal scheduling in IoT systems with Markov channels by formulating a partially observable restless bandit problem and proposing index-based policies that outperform baselines.

Contribution

It introduces a novel approach to age-optimal scheduling over Markov channels using Whittle's index in a partially observable setting, with closed-form solutions for low complexity.

Findings

Proposed policies outperform baseline methods in simulations.

Policies are effective under limited resources and large network sizes.

Established indexability and derived closed-form Whittle-like index.

Abstract

There is a surge of need for fresh information with the overwhelming proliferation of the Internet of Things (IoT) applications. To characterize the information freshness perceived by the destination, the age of information (AoI) has been proposed. In this paper, we consider an IoT system with multiple devices sending status update packets to a central controller through time-correlated Markov channels and assume that the instantaneous channel states are not available to the central controller before making scheduling decisions. To ensure information freshness, we investigate a timely scheduling problem that minimizes the total expected time-average AoI under a strict communications bandwidth constraint. We formulate this problem as a partially observable restless multi-armed bandit problem. Using Lagrangian relaxation, we decouple the relaxed problem into multiple sub-problems and prove the threshold structure of their optimal policies. Armed with this property, we establish the indexability for the decoupled problem and design an algorithm to compute the Whittle's index. To reduce implementation complexity, we further derive the Whittle-like index in closed-form for low-complexity scheduling. Simulation results show that the proposed index-based policies outperform the baselines, remain close to the optimal policy or relaxed lower bound, and are especially effective when scheduling resources are limited or the network size is large.