Efficient Solving of Large Single Input Superstate Decomposable Markovian Decision Process

TL;DR

This paper introduces a scalable, exact policy evaluation method for a structured class of large MDPs called SISDMDPs, leveraging structural decompositions to improve efficiency in solving infinite-horizon problems.

Contribution

It extends aggregation techniques to SISDMDPs, enabling efficient policy evaluation in large, structured MDPs with applications to average and discounted rewards.

Findings

Developed an exact policy evaluation method for SISDMDPs.

Achieved scalable solutions for large state spaces.

Applicable to both average and discounted reward MDPs.

Abstract

Solving Markov Decision Processes (MDPs) remains a central challenge in sequential decision-making, especially when dealing with large state spaces and long-term optimization criteria. A key step in Bellman dynamic programming algorithms is the policy evaluation, which becomes computationally demanding in infinite-horizon settings such as average-reward or discounted-reward formulations. In the context of Markov chains, aggregation and disaggregation techniques have for a long time been used to reduce complexity by exploiting structural decompositions. In this work, we extend these principles to a structured class of MDPs. We define the Single-Input Superstate Decomposable Markov Decision Process (SISDMDP), which combines Chiu's single-input decomposition with Robertazzi's single-cycle recurrence property. When a policy induces this structure, the resulting transition graph can be decomposed into interacting components with centralized recurrence. We develop an exact and efficient policy evaluation method based on this structure. This yields a scalable solution applicable to both average and discounted reward MDPs.