Adaptive Scheduling: A Reinforcement Learning Whittle Index Approach for Wireless Sensor Networks

TL;DR

This paper introduces WIQL-UCB, a reinforcement learning framework for scheduling in wireless sensor networks that is hyperparameter-free, computationally efficient, and achieves near-optimal performance across diverse RMAB problems.

Contribution

It presents a novel Whittle Index Q-Learning method with UCB exploration that removes the need for problem-specific tuning, enhancing generalizability and efficiency.

Findings

Achieves near-optimal performance on RMAB benchmarks and sensor scheduling tasks.

Requires significantly less memory and computation compared to deep RL methods.

Outperforms existing non-Whittle and Whittle-based baselines across various settings.

Abstract

We propose a reinforcement learning based scheduling framework for Restless Multi-Armed Bandit (RMAB) problems, centred on a Whittle Index Q-Learning policy with Upper Confidence Bound (UCB) exploration, referred to as WIQL-UCB. Unlike existing approaches that rely on fixed or adaptive epsilon-greedy strategies and require careful hyperparameter tuning, the proposed method removes problem-specific tuning and is therefore more generalisable across diverse RMAB settings. We evaluate WIQL-UCB on standard RMAB benchmarks and on a practical sensor scheduling application based on the Age of Incorrect Information (AoII), using an edge-based state estimation scheme that requires no prior knowledge of system dynamics. Experimental results show that WIQL-UCB achieves near-optimal performance while significantly improving computational and memory efficiency. For a representative problem size of N = 15 and M = 3, the proposed method requires only around 600 bytes of memory, compared with several kilobytes for tabular Q-learning and hundreds of kilobytes to megabytes for deep reinforcement learning baselines. In addition, WIQL-UCB achieves sub-millisecond per-decision runtimes and is several times faster than deep reinforcement learning approaches, while maintaining competitive performance. Overall, these results demonstrate that WIQL-UCB consistently outperforms both non-Whittle-based and Whittle-index learning baselines across a wide range of RMAB settings.