Partially observed controlled Markov chains and optimal control of the Wonham filter

TL;DR

This paper studies optimal control of partially observed Markov chains, transforming the problem into a separated control problem involving the Wonham filter, and establishes theoretical results including viscosity solutions and maximum principles.

Contribution

It introduces a novel construction of controlled Markov chains with stochastic transition rates and proves the equivalence of the original and separated control problems using the Wonham filter.

Findings

Characterizes the value function as a viscosity solution to the dynamic programming equations.

Proves the equivalence between the original and separated control problems.

Provides verification theorems and necessary conditions for optimality.

Abstract

We consider a class of optimal control problems, with finite or infinite horizon, for a continuous-time Markov chain with finite state space. In this case, the control process affects the transition rates. We suppose that the controlled process can not be observed, and at any time the control actions are chosen based on the observation of a related stochastic process perturbed by an exogenous Brownian motion. We describe a construction of the controlled Markov chain, having stochastic transition rates adapted to the observation filtration. By a change of probability measure of Girsanov type, we introduce the so-called separated optimal control problem, where the state is the conditional (unnormalized) distribution of the controlled Markov chain and the observation process becomes a driving Brownian motion, and we prove the equivalence with the original control problem. The controlled equations for the separated problem are an instance of the Wonham filtering equations. Next we present an analysis of the separated problem: we characterize the value function as the unique viscosity solution to the dynamic programming equations (both in the parabolic and the elliptic case) we prove verifications theorems and a version of the stochastic maximum principle in the form of a necessary conditions for optimality.