Constrained discounted Markov decision processes with Borel state spaces

TL;DR

This paper investigates constrained discounted Markov decision processes on Borel spaces with unbounded rewards, establishing conditions for the existence of stationary and deterministic optimal policies, and illustrating results with five examples.

Contribution

It provides new solvability results for CMDPs with unbounded rewards on Borel spaces, including conditions for deterministic policies under non-atomic initial distributions.

Findings

Optimal policies exist within stationary randomized policies.

Deterministic policies are achievable under non-atomic initial distributions.

Theoretical results are demonstrated through five illustrative examples.

Abstract

We study discrete-time discounted constrained Markov decision processes (CMDPs) on Borel spaces with unbounded reward functions. In our approach the transition probability functions are weakly or set-wise continuous. The reward functions are upper semicontinuous in state-action pairs or semicontinuous in actions. Our aim is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of stationary randomized policies. Then, we indicate that if the initial distribution and transition probabilities are non-atomic, then using a general purification result of Feinberg and Piunovskiy, stationary optimal policies can be deterministic. Our main results are illustrated by five examples.