Time Consistency for Multistage Stochastic Optimization Problems under Constraints in Expectation

TL;DR

This paper introduces a new concept of time consistent solutions for multistage stochastic optimization problems with expectation constraints, linking it to Markov decision processes and demonstrating its application in a dam management scenario.

Contribution

It defines a novel notion of time consistency for multistage stochastic problems with constraints, connecting it to state variables in Markov decision processes, and provides a finite-dimensional solution approach.

Findings

Time consistent solutions can be characterized using finite-dimensional state variables.

The proposed framework applies to problems with stagewise independent noise processes.

Illustration provided through a dam management example.

Abstract

We consider sequences-indexed by time (discrete stages)-of families of multistage stochastic optimization problems. At each time, the optimization problems in a family are parameterized by some quantities (initial states, constraint levels.. .). In this framework, we introduce an adapted notion of time consistent optimal solutions, that is, solutions that remain optimal after truncation of the past and that are optimal for any values of the parameters. We link this time consistency notion with the concept of state variable in Markov Decision Processes for a class of multistage stochastic optimization problems incorporating state constraints at the final time, either formulated in expectation or in probability. For such problems, when the primitive noise random process is stagewise independent and takes a finite number of values, we show that time consistent solutions can be obtained by considering a finite dimensional state variable. We illustrate our results on a simple dam management problem.