Deep neural networks can provably solve Bellman equations for Markov decision processes without the curse of dimensionality

TL;DR

This paper demonstrates that deep neural networks can efficiently approximate solutions to Bellman equations in high-dimensional Markov decision processes, avoiding the curse of dimensionality under certain approximation conditions.

Contribution

The authors establish a polynomial growth bound on neural network parameters for approximating Bellman equation solutions in high-dimensional MDPs, using the MLFP scheme.

Findings

Neural networks can approximate Q-functions with polynomially growing parameters.

The approach avoids the curse of dimensionality in solving Bellman equations.

The method applies to MDPs with infinite horizon and finite control sets.

Abstract

Discrete time stochastic optimal control problems and Markov decision processes (MDPs) are fundamental models for sequential decision-making under uncertainty and as such provide the mathematical framework underlying reinforcement learning theory. A central tool for solving MDPs is the Bellman equation and its solution, the so-called $Q$-function. In this article, we construct deep neural network (DNN) approximations for $Q$-functions associated to MDPs with infinite time horizon and finite control set $A$. More specifically, we show that if the the payoff function and the random transition dynamics of the MDP can be suitably approximated by DNNs with leaky rectified linear unit (ReLU) activation, then the solutions $Q_d\colon \mathbb R^d\to \mathbb R^{|A|}$, $d\in \mathbb{N}$, of the associated Bellman equations can also be approximated in the $L^2$-sense by DNNs with leaky ReLU activation whose numbers of parameters grow at most polynomially in both the dimension $d\in \mathbb{N}$ of the state space and the reciprocal $1/\varepsilon$ of the prescribed error $\varepsilon\in (0,1)$. Our proof relies on the recently introduced full-history recursive multilevel fixed-point (MLFP) approximation scheme.