Dynamic programming and dimensionality in convex stochastic optimization and control

TL;DR

This paper explores how certain stochastic control problems can be formulated to reduce the dimensionality of the Bellman equations, improving the efficiency of dynamic programming algorithms.

Contribution

It introduces a new formulation of stochastic control problems that allows for lower-dimensional Bellman equations, with general existence results and analysis of Markovian cases.

Findings

Dimensionality reduction in Bellman equations for specific control structures

Existence of solutions without compactness assumptions

Dimensionality reduction in Markovian cases

Abstract

This paper studies stochastic optimization problems and associated Bellman equations in formats that allow for reduced dimensionality of the cost-to-go functions. In particular, we study stochastic control problems in the ``decision-hazard-decision'' form where at each stage, the system state is controlled both by predictable as well as adapted controls. Such an information structure may result in a lower dimensional system state than what is required in more traditional ``decision-hazard'' or ``hazard-decision'' formulations. The dimension is critical for the complexity of numerical dynamic programming algorithms and, in particular, for cutting plane schemes such as the stochastic dual dynamic programming algorithm. Our main result characterizes optimal solutions and optimum values in terms of solutions to generalized Bellman equations. Existence of solutions to the Bellman equations is established under general conditions that do not require compactness. We allow for general randomness but show that, in the Markovian case, the dimensionality of the Bellman equations reduces with respect to randomness just like in more traditional control formulations.