Optimal control of semi-Markov processes with a backward stochastic differential equations approach

TL;DR

This paper introduces a novel approach using backward stochastic differential equations (BSDEs) to solve optimal control problems for semi-Markov processes, establishing a new link between BSDEs and semi-Markov control theory.

Contribution

It is the first to employ BSDEs for semi-Markov process control, deriving a new integral form of the HJB equation and proving the BSDE solution's uniqueness and relation to the value function.

Findings

BSDEs represent the value function and optimal control law.

The HJB equation is characterized by an additional differential term $ extpartial_a$.

The BSDE provides the unique classical solution to the HJB equation.

Abstract

In the present work we employ, for the first time, backward stochastic differential equations (BSDEs) to study the optimal control of semi-Markov processes on finite horizon, with general state and action spaces. More precisely, we prove that the value function and the optimal control law can be represented by means of the solution of a class of BSDEs driven by a semi-Markov process or, equivalently, by the associated random measure. The peculiarity of the semi-Markov framework, with respect to the pure jump Markov case, consists in the proof of the relation between BSDE and optimal control problem. This is done, as usual, via the Hamilton-Jacobi-Bellman (HJB) equation, which however in the semi-Markov case is characterized by an additional differential term $\partial_a$. Taking into account the particular structure of semi-Markov processes we rewrite the HJB equation in a suitable integral form which involves a directional derivative operator $D$ related to $\partial_a$. Then, using a formula of Ito type tailor-made for semi-Markov processes and the operator $D$, we are able to prove that the BSDE provides the unique classical solution to the HJB equation, which is shown to be the value function of our control problem.