Networked Restless Multi-Arm Bandits with Reinforcement Learning

TL;DR

This paper introduces Networked RMAB, integrating RMAB with network interactions, and develops an efficient Q-learning algorithm that outperforms existing methods on real-world data.

Contribution

It presents a novel Networked RMAB framework, establishes its theoretical properties, and develops a scalable Q-learning algorithm for networked environments.

Findings

Q-learning outperforms $k$-step look-ahead methods.

Network effects significantly improve decision-making.

Theoretical guarantees ensure convergence of the approximation.

Abstract

Restless Multi-Armed Bandits (RMABs) are a powerful framework for sequential decision-making, widely applied in resource allocation and intervention optimization challenges in public health. However, traditional RMABs assume independence among arms, limiting their ability to account for interactions between individuals that can be common and significant in a real-world environment. This paper introduces Networked RMAB, a novel framework that integrates the RMAB model with the independent cascade model to capture interactions between arms in networked environments. We define the Bellman equation for networked RMAB and present its computational challenge due to exponentially large action and state spaces. To resolve the computational challenge, we establish the submodularity of Bellman equation and apply the hill-climbing algorithm to achieve a $1-\frac{1}{e}$ approximation guarantee in Bellman updates. Lastly, we prove that the approximate Bellman updates are guaranteed to converge by a modified contraction analysis. We experimentally verify these results by developing an efficient Q-learning algorithm tailored to the networked setting. Experimental results on real-world graph data demonstrate that our Q-learning approach outperforms both $k$-step look-ahead and network-blind approaches, highlighting the importance of capturing and leveraging network effects where they exist.