Optimal control of continuous-time Markov chains with noise-free observation

TL;DR

This paper addresses an optimal control problem for continuous-time Markov chains with noise-free partial observations, transforming it into a complete observation problem and characterizing the value function through fixed point and viscosity solution methods.

Contribution

It introduces a novel approach to handle noise-free partial observations by linking the original and separated control problems and characterizing the value function via fixed point and viscosity solutions.

Findings

Established a connection between the original and separated control problems.

Proved the value function as a fixed point of a contraction mapping.

Demonstrated the value function as a viscosity solution of a HJB equation.

Abstract

We consider an infinite horizon optimal control problem for a continuous-time Markov chain $X$ in a finite set $I$ with noise-free partial observation. The observation process is defined as $Y_t = h(X_t)$, $t \geq 0$, where $h$ is a given map defined on $I$. The observation is noise-free in the sense that the only source of randomness is the process $X$ itself. The aim is to minimize a discounted cost functional and study the associated value function $V$. After transforming the control problem with partial observation into one with complete observation (the separated problem) using filtering equations, we provide a link between the value function $v$ associated to the latter control problem and the original value function $V$. Then, we present two different characterizations of $v$ (and indirectly of $V$): on one hand as the unique fixed point of a suitably defined contraction mapping and on the other hand as the unique constrained viscosity solution (in the sense of Soner) of a HJB integro-differential equation. Under suitable assumptions, we finally prove the existence of an optimal control.