A Federated Online Restless Bandit Framework for Cooperative Resource Allocation

TL;DR

This paper introduces a federated learning framework for cooperative resource allocation in unknown Markov reward processes, proposing a novel algorithm with efficiency, privacy, and convergence guarantees, validated through multi-user multi-channel access experiments.

Contribution

It presents a federated online RMAB framework and a FedTSWI algorithm for unknown dynamics, combining federated learning with Thompson Sampling for efficient, private multi-agent resource allocation.

Findings

Achieves fast convergence rate of O(√T log T).

Reduces sample complexity as the number of agents increases.

Outperforms baseline algorithms in experiments.

Abstract

Restless multi-armed bandits (RMABs) have been widely utilized to address resource allocation problems with Markov reward processes (MRPs). Existing works often assume that the dynamics of MRPs are known prior, which makes the RMAB problem solvable from an optimization perspective. Nevertheless, an efficient learning-based solution for RMABs with unknown system dynamics remains an open problem. In this paper, we study the cooperative resource allocation problem with unknown system dynamics of MRPs. This problem can be modeled as a multi-agent online RMAB problem, where multiple agents collaboratively learn the system dynamics while maximizing their accumulated rewards. We devise a federated online RMAB framework to mitigate the communication overhead and data privacy issue by adopting the federated learning paradigm. Based on this framework, we put forth a Federated Thompson Sampling-enabled Whittle Index (FedTSWI) algorithm to solve this multi-agent online RMAB problem. The FedTSWI algorithm enjoys a high communication and computation efficiency, and a privacy guarantee. Moreover, we derive a regret upper bound for the FedTSWI algorithm. Finally, we demonstrate the effectiveness of the proposed algorithm on the case of online multi-user multi-channel access. Numerical results show that the proposed algorithm achieves a fast convergence rate of $\mathcal{O}(\sqrt{T\log(T)})$ and better performance compared with baselines. More importantly, its sample complexity decreases with the number of agents.