A dynamic analytic method for risk-aware controlled martingale problems

TL;DR

This paper introduces a new method for solving risk-aware control problems using a dynamic, nonlinear programming approach based on martingale problem formulations, applicable to general state and action spaces.

Contribution

It develops a tractable, equivalent nonlinear programming formulation for risk-aware control problems with martingale dynamics, extending analysis to general Polish spaces and relaxed controls.

Findings

The method provides an explicit construction of optimal controls.

The formulations are proven equivalent and optimal controls can be Markov.

An example demonstrates the practical application of the approach.

Abstract

We present a new, tractable method for solving and analyzing risk-aware control problems over finite and infinite, discounted time-horizons where the dynamics of the controlled process are described as a martingale problem. Supposing general Polish state and action spaces, and using generalized, relaxed controls, we state a risk-aware dynamic optimal control problem of minimizing risk of costs described by a generic risk function. We then construct an alternative formulation that takes the form of a nonlinear programming problem, constrained by the dynamic, {i.e.} time-dependent, and linear Kolmogorov forward equation describing the distribution of the state and accumulated costs. We show that the formulations are equivalent, and that the optimal control process can be taken to be Markov in the controlled process state, running costs, and time. We further prove that under additional conditions, the optimal value is attained. An example numeric problem is presented and solved.