Subdifferentials of Value Functions in Nonconvex Dynamic Programming for Nonstationary Stochastic Processes

TL;DR

This paper develops a variational analysis framework to compute subdifferentials of value functions in nonconvex, nonstationary stochastic dynamic programming within Banach spaces, enabling new optimality conditions without convexity.

Contribution

It introduces a method to derive subgradients and optimality conditions for nonconvex, nonstationary stochastic DP in Banach spaces, extending beyond traditional convex assumptions.

Findings

Derived subdifferential formulas for value functions in $L^p$ spaces.

Established verifiable conditions for value function smoothness.

Provided necessary optimality conditions via stochastic Euler inclusion.

Abstract

The main goal of this paper is to apply the machinery of variational analysis and generalized differentiation to study infinite horizon stochastic dynamic programming (DP) with discrete time in the Banach space setting without convexity assumptions. Unlike to standard stochastic DP with stationary Markov processes, we investigate here stochastic DP in $L^p$ spaces to deal with nonstationary stochastic processes, which describe a more flexible learning procedure for the decision-maker. Our main concern is to calculate generalized subgradients of the corresponding value function and to derive necessary conditions for optimality in terms of the stochastic Euler inclusion under appropriate Lipschitzian assumptions. The usage of the subdifferential formula for integral functionals on $L^p$ spaces allows us, in particular, to find verifiable conditions to ensure smoothness of the value function without any convexity and/or interiority assumptions.