Exact finite approximations of average-cost countable Markov Decision Processes

TL;DR

This paper introduces a method to embed countable-state Markov decision processes into finite-state processes, preserving optimal costs and dynamics, facilitating easier computation and implementation.

Contribution

The paper presents a novel embedding technique that creates finite-state approximations of countable-state Markov decision processes while maintaining key properties.

Findings

Finite embedded processes have the same optimal cost as original processes.

Embedded processes preserve the original dynamics within the approximating set.

The method simplifies computation and implementation for complex MDPs.

Abstract

For a countable-state Markov decision process we introduce an embedding which produces a finite-state Markov decision process. The finite-state embedded process has the same optimal cost, and moreover, it has the same dynamics as the original process when restricting to the approximating set. The embedded process can be used as an approximation which, being finite, is more convenient for computation and implementation.