Continuous-time Markov Decision Processes with Finite-horizon Expected Total Cost Criteria

TL;DR

This paper establishes the existence of optimal policies for finite-horizon continuous-time Markov decision processes with unbounded rates, using occupation measures and linear programming techniques, applicable to both constrained and unconstrained cases.

Contribution

It introduces new conditions for optimal policy existence in unbounded rate CTMDPs and employs occupation measures and linear programming for constrained problems.

Findings

Existence of optimal deterministic Markov policies for unconstrained cases.

Optimal constrained policies can be obtained via linear programming.

The approach applies to systems with unbounded transition and cost rates.

Abstract

This paper deals with the unconstrained and constrained cases for continuous-time Markov decision processes under the finite-horizon expected total cost criterion. The state space is denumerable and the transition and cost rates are allowed to be unbounded from above and from below. We give conditions for the existence of optimal policies in the class of all randomized history-dependent policies. For the unconstrained case, using the analogue of the forward Kolmogorov equation in the form of conditional expectation, we show that the finite-horizon optimal value function is the unique solution to the optimality equation and obtain the existence of an optimal deterministic Markov policy. For the constrained case, employing the technique of occupation measures, we first give an equivalent characterization of the occupation measures, and derive that for each occupation measure generated by a randomized history-dependent policy, there exists an occupation measure generated by a randomized Markov policy equal to it. Then using the compactness and convexity of the set of all occupation measures, we obtain the existence of a constrained-optimal randomized Markov policy. Moreover, the constrained optimization problem is reformulated as a linear program, and the strong duality between the linear program and its dual program is established. Finally, a controlled birth and death system is used to illustrate our main results.