Decision-Focused Evaluation: Analyzing Performance of Deployed Restless Multi-Arm Bandits

TL;DR

This paper analyzes the real-world deployment of restless multi-arm bandit systems in public health, revealing that higher prediction accuracy does not always lead to better system performance, and introduces decision-focused evaluation metrics.

Contribution

It provides the first analysis of deployed RMAB systems in public health, highlighting the complex relationship between prediction accuracy and system performance, and proposes decision-focused evaluation metrics.

Findings

Improvement in prediction accuracy can degrade overall system performance.

Decision-focused metrics better explain system performance than traditional accuracy measures.

Deployed RMAB systems in public health reveal non-linear effects of prediction improvements.

Abstract

Restless multi-arm bandits (RMABs) is a popular decision-theoretic framework that has been used to model real-world sequential decision making problems in public health, wildlife conservation, communication systems, and beyond. Deployed RMAB systems typically operate in two stages: the first predicts the unknown parameters defining the RMAB instance, and the second employs an optimization algorithm to solve the constructed RMAB instance. In this work we provide and analyze the results from a first-of-its-kind deployment of an RMAB system in public health domain, aimed at improving maternal and child health. Our analysis is focused towards understanding the relationship between prediction accuracy and overall performance of deployed RMAB systems. This is crucial for determining the value of investing in improving predictive accuracy towards improving the final system performance, and is useful for diagnosing, monitoring deployed RMAB systems. Using real-world data from our deployed RMAB system, we demonstrate that an improvement in overall prediction accuracy may even be accompanied by a degradation in the performance of RMAB system -- a broad investment of resources to improve overall prediction accuracy may not yield expected results. Following this, we develop decision-focused evaluation metrics to evaluate the predictive component and show that it is better at explaining (both empirically and theoretically) the overall performance of a deployed RMAB system.