Gradual-impulsive control for continuous-time Markov decision processes with total undiscounted costs and constraints: linear programming approach via a reduction method

TL;DR

This paper introduces a linear programming approach for solving constrained continuous-time Markov decision processes with both gradual and impulsive controls, by reducing the problem to a simpler model with only gradual control.

Contribution

It proposes a novel reduction method that simplifies the constrained control problem, enabling an effective linear programming solution for models with total undiscounted costs.

Findings

Reduction method simplifies the original model

Linear programming approach solves the constrained problem

Addresses an open issue in the literature

Abstract

We consider the constrained optimal control problem for the gradual-impulsive CTMDP model with the performance criteria being the expected total undiscounted costs (from the running cost and the cost from each time an impulse being applied). The discounted model is covered as a special case. We justify fully a reduction method, and close an open issue in the previous literature. The reduction method induces an equivalent but simpler standard CTMDP model with gradual control only, based on which, we establish effectively, under rather natural conditions, a linear programming approach for solving the concerned constrained optimal control problem.