A Uniform-grid Discretization Algorithm for Stochastic Control with Risk Constraints

TL;DR

This paper introduces a discretization algorithm for risk-constrained dynamic programming in stochastic control, addressing computational challenges by discretizing continuous spaces and providing error bounds for the approximation.

Contribution

It proposes a finite space approximation method for risk-constrained dynamic programming, with proven linear error bounds related to discretization step size.

Findings

Discretization effectively approximates the Bellman operator in risk-constrained problems.

Error bounds of the approximation are linearly related to discretization step size.

Implementation details and potential modifications are discussed.

Abstract

In this paper, we present a discretization algorithm for finite horizon risk constrained dynamic programming algorithm in [Chow_Pavone_13]. Although in a theoretical standpoint, Bellman's recursion provides a systematic way to find optimal value functions and generate optimal history dependent policies, there is a serious computational issue. Even if the state space and action space of this constrained stochastic optimal control problem are finite, the spaces of risk threshold and the feasible risk update are closed bounded subset of real numbers. This prohibits any direct applications of unconstrained finite state iterative methods in dynamic programming found in [Bertsekas_05]. In order to approximate Bellman's operator derived in [Chow_Pavone_13], we discretize the continuous action spaces and formulate a finite space approximation for the exact dynamic programming algorithm. We will also prove that the approximation error bound of optimal value functions is bound linearly by the step size of discretization. Finally, details for implementations and possible modifications are discussed.