Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization

TL;DR

This paper introduces PreFeRMAB, a neural network-based pre-trained model for restless multi-arm bandits that generalizes zero-shot to unseen problems, handles continuous states, and improves sample efficiency for fine-tuning.

Contribution

The paper presents a novel pre-trained neural network model for RMABs that generalizes across problem instances, supports multi-action and continuous states, and includes a new update rule with convergence guarantees.

Findings

PreFeRMAB achieves zero-shot generalization to unseen RMABs.

The model improves sample efficiency in fine-tuning on specific instances.

Empirical results demonstrate advantages on real-world inspired problems.

Abstract

Restless multi-arm bandits (RMABs), a class of resource allocation problems with broad application in areas such as healthcare, online advertising, and anti-poaching, have recently been studied from a multi-agent reinforcement learning perspective. Prior RMAB research suffers from several limitations, e.g., it fails to adequately address continuous states, and requires retraining from scratch when arms opt-in and opt-out over time, a common challenge in many real world applications. We address these limitations by developing a neural network-based pre-trained model (PreFeRMAB) that has general zero-shot ability on a wide range of previously unseen RMABs, and which can be fine-tuned on specific instances in a more sample-efficient way than retraining from scratch. Our model also accommodates general multi-action settings and discrete or continuous state spaces. To enable fast generalization, we learn a novel single policy network model that utilizes feature information and employs a training procedure in which arms opt-in and out over time. We derive a new update rule for a crucial $\lambda$-network with theoretical convergence guarantees and empirically demonstrate the advantages of our approach on several challenging, real-world inspired problems.