Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits

TL;DR

This paper introduces BCoR, a Bayesian online reinforcement learning method for resource allocation in public health, effectively handling complex, non-stationary environments to improve intervention outcomes in underserved communities.

Contribution

The paper presents BCoR, a novel Bayesian Thompson sampling approach for contextual restless bandits, specifically designed for public health applications with limited data and dynamic settings.

Findings

BCoR outperforms existing methods in finite-sample scenarios.

BCoR demonstrates practical utility with real-world maternal health data.

The approach effectively models context and non-stationarity in public health interventions.

Abstract

Public health programs often provide interventions to encourage program adherence, and effectively allocating interventions is vital for producing the greatest overall health outcomes, especially in underserved communities where resources are limited. Such resource allocation problems are often modeled as restless multi-armed bandits (RMABs) with unknown underlying transition dynamics, hence requiring online reinforcement learning (RL). We present Bayesian Learning for Contextual RMABs (BCoR), an online RL approach for RMABs that novelly combines techniques in Bayesian modeling with Thompson sampling to flexibly model the complex RMAB settings present in public health program adherence problems, namely context and non-stationarity. BCoR's key strength is the ability to leverage shared information within and between arms to learn the unknown RMAB transition dynamics quickly in intervention-scarce settings with relatively short time horizons, which is common in public health applications. Empirically, BCoR achieves substantially higher finite-sample performance over a range of experimental settings, including a setting using real-world adherence data that was developed in collaboration with ARMMAN, an NGO in India which runs a large-scale maternal mHealth program, showcasing BCoR practical utility and potential for real-world deployment.