The implications of state aggregation in deteriorating Markov Decision Processes with optimal threshold policies

TL;DR

This paper examines how state aggregation in Markov Decision Processes affects the optimality of threshold policies, highlighting trade-offs between model precision and data limitations through theoretical analysis and simulations.

Contribution

It provides conditions under which aggregated MDPs retain threshold policies and analyzes the trade-offs in policy quality with different levels of state aggregation.

Findings

Larger state spaces offer greater potential for policy improvement.

Aggregated MDPs are preferable when data is limited.

Threshold policies can be preserved under certain aggregation conditions.

Abstract

Markov Decision Processes (MDPs) are mathematical models of sequential decision-making under uncertainty that have found applications in healthcare, manufacturing, logistics, and others. In these models, a decision-maker observes the state of a stochastic process and determines which action to take with the goal of maximizing the expected total discounted rewards received. In many applications, the state space of the true system is large and there may be limited observations out of certain states to estimate the transition probability matrix. To overcome this, modelers will aggregate the true states into ``superstates" resulting in a smaller state space. This aggregation process improves computational tractability and increases the number of observations among superstates. Thus, the modeler's choice of state space leads to a trade-off in transition probability estimates. While coarser discretization of the state space gives more observations in each state to estimate the transition probability matrix, this comes at the cost of precision in the state characterization and resulting policy recommendations. In this paper, we consider the implications of this modeling decision on the resulting policies from MDPs for which the true model is expected to have a threshold policy that is optimal. We analyze these MDPs and provide conditions under which the aggregated MDP will also have an optimal threshold policy. Using a simulation study, we explore the trade-offs between more fine and more coarse aggregation. We explore the the show that there is the highest potential for policy improvement on larger state spaces, but that aggregated MDPs are preferable under limited data. We discuss how these findings the implications of our findings for modelers who must select which state space design to use.