Complexity of Stochastic Dual Dynamic Programming

TL;DR

This paper analyzes the convergence rates of stochastic dual dynamic programming (SDDP), providing iteration complexity bounds for both deterministic and stochastic variants, and highlighting their efficiency for multi-stage stochastic optimization with many stages.

Contribution

The paper introduces novel mathematical tools to establish iteration complexity bounds for SDDP and its variants, advancing theoretical understanding of their convergence behavior.

Findings

Iteration complexity increases linearly with the number of stages for discounted problems.

Deterministic variants are efficient for problems with many stages and few decision variables.

Results are relevant to reinforcement learning and stochastic control without discretizing state spaces.

Abstract

Stochastic dual dynamic programming is a cutting plane type algorithm for multi-stage stochastic optimization originated about 30 years ago. In spite of its popularity in practice, there does not exist any analysis on the convergence rates of this method. In this paper, we first establish the number of iterations, i.e., iteration complexity, required by a basic dynamic cutting plane method for solving relatively simple multi-stage optimization problems, by introducing novel mathematical tools including the saturation of search points. We then refine these basic tools and establish the iteration complexity for both deterministic and stochastic dual dynamic programming methods for solving more general multi-stage stochastic optimization problems under the standard stage-wise independence assumption. Our results indicate that the complexity of some deterministic variants of these methods mildly increases with the number of stages $T$, in fact linearly dependent on $T$ for discounted problems. Therefore, they are efficient for strategic decision making which involves a large number of stages, but with a relatively small number of decision variables in each stage. Without explicitly discretizing the state and action spaces, these methods might also be pertinent to the related reinforcement learning and stochastic control areas.