Cascade Markov Decision Processes: Theory and Applications

TL;DR

This paper introduces the theory of cascade Markov decision processes, analyzing optimal control strategies for time-varying Markov chains with Markovian transition rates, with applications in finance and decision making.

Contribution

It develops a new theoretical framework for cascade Markov decision processes and demonstrates solution techniques for complex control problems.

Findings

Differential equations determine optimal controls in some cases

Singular control problems arise in more complex scenarios

Applications include finance and behavioral decision making

Abstract

This paper considers the optimal control of time varying continuous time Markov chains whose transition rates are themselves Markov processes. In one set of problems the solution of an ordinary differential equation is shown to determine the optimal performance and feedback controls, while some other cases are shown to lead to singular optimal control problems which are more difficult to solve. Solution techniques are demonstrated using examples from finance to behavioral decision making.