Stochastic Optimal Control With Dynamic, Time-Consistent Risk Constraints

TL;DR

This paper introduces a dynamic programming method for solving stochastic optimal control problems with dynamic, time-consistent risk constraints, advancing the integration of risk metrics that ensure rational risk preferences over multiple periods.

Contribution

It formulates a new class of stochastic control problems with time-consistent risk constraints and develops a dynamic programming solution for these problems.

Findings

The approach computes optimal costs via value iteration.

The method generalizes to various risk metrics and control policies.

A simple example illustrates the solution process.

Abstract

In this paper we present a dynamic programing approach to stochastic optimal control problems with dynamic, time-consistent risk constraints. Constrained stochastic optimal control problems, which naturally arise when one has to consider multiple objectives, have been extensively investigated in the past 20 years, however, in most formulations, the constraints are formulated as either risk-neutral (i.e., by considering an expected cost), or by applying static, single-period risk metrics with limited attention to "time-consistency" (i.e., to whether such metrics ensure rational consistency of risk preferences across multiple periods). Recently, significant strides have been made in the development of a rigorous theory of dynamic, \emph{time-consistent} risk metrics for multi-period (risk-sensitive) decision processes, however, their integration within constrained stochastic optimal control problems has received little attention. The goal of this paper is to bridge this gap. First, we formulate the stochastic optimal control problem with dynamic, time-consistent risk constraints and we characterize the tail subproblems (which requires the addition of a Markovian structure to the risk metrics). Second, we develop a dynamic programming approach for its solution, which allows to compute the optimal costs by value iteration. Finally, we discuss both theoretical and practical features of our approach, such as generalizations, construction of optimal control policies, and computational aspects. A simple, two-state example is given to illustrate the problem setup and the solution approach.