State Discretization for Continuous-State MDPs in Infectious Disease Control

TL;DR

This paper introduces a novel discretization algorithm for continuous-state Markov decision processes, enabling more effective infectious disease control policies, demonstrated through COVID-19 case studies.

Contribution

The paper proposes a new state discretization method tailored for continuous-state MDPs in infectious disease management, improving policy approximation accuracy.

Findings

Discretization improves policy performance over uniform methods.

Algorithm performs well on synthetic and real COVID-19 data.

Enhanced control strategies reduce disease prevalence effectively.

Abstract

Repeated decision-making problems under uncertainty may arise in the health policy context, such as infectious disease control for COVID-19 and other epidemics. These problems may sometimes be effectively solved using Markov decision processes (MDPs). However, the continuous or large state space of such problems for capturing infectious disease prevalence renders it difficult to implement tractable MDPs to identify the optimal disease control policy over time. We therefore develop an algorithm for discretizing continuous states for approximate MDP solutions in this context. We benchmark performance against a uniform discretization using both a synthetic example and an example of COVID-19 in Los Angeles County.