Constrained total undiscounted continuous-time Markov decision processes

TL;DR

This paper addresses constrained optimal control in continuous-time Markov decision processes with total undiscounted criteria, establishing existence of optimal policies and reducing the problem to a discrete-time model under broad conditions.

Contribution

It proves the existence of optimal stationary policies for constrained undiscounted CTMDPs and justifies model reduction to DTMDPs despite unbounded transition rates and non-absorbing processes.

Findings

Existence of optimal stationary policies under standard conditions

Reduction of CTMDP to DTMDP model for analysis

Handling unbounded transition rates and non-absorbing processes

Abstract

The present paper considers the constrained optimal control problem with total undiscounted criteria for a continuous-time Markov decision process (CTMDP) in Borel state and action spaces. Under the standard compactness and continuity conditions, we show the existence of an optimal stationary policy out of the class of general nonstationary ones. In the process, we justify the reduction of the CTMDP model to a discrete-time Markov decision process (DTMDP) model based on the studies of the undiscounted occupancy and occupation measures. We allow that the controlled process is not necessarily absorbing, and the transition rates are not necessarily separated from zero, and can be arbitrarily unbounded; these features count for the main technical difficulties in studying undiscounted CTMDP models.