Sample Complexity of Neural Policy Mirror Descent for Policy Optimization on Low-Dimensional Manifolds

TL;DR

This paper analyzes the sample complexity of neural policy mirror descent in high-dimensional reinforcement learning, showing it exploits low-dimensional manifold structures to improve efficiency.

Contribution

It provides the first theoretical analysis demonstrating how NPMD leverages low-dimensional structures to overcome the curse of dimensionality in policy optimization.

Findings

NPMD can find an ε-optimal policy with sample complexity depending on the intrinsic dimension d.

CNN approximation errors are controlled by network size and inherited smoothness.

Results explain the empirical success of deep policy gradient methods in high-dimensional spaces.

Abstract

Policy gradient methods equipped with deep neural networks have achieved great success in solving high-dimensional reinforcement learning (RL) problems. However, current analyses cannot explain why they are resistant to the curse of dimensionality. In this work, we study the sample complexity of the neural policy mirror descent (NPMD) algorithm with deep convolutional neural networks (CNN). Motivated by the empirical observation that many high-dimensional environments have state spaces possessing low-dimensional structures, such as those taking images as states, we consider the state space to be a $d$-dimensional manifold embedded in the $D$-dimensional Euclidean space with intrinsic dimension $d\ll D$. We show that in each iteration of NPMD, both the value function and the policy can be well approximated by CNNs. The approximation errors are controlled by the size of the networks, and the smoothness of the previous networks can be inherited. As a result, by properly choosing the network size and hyperparameters, NPMD can find an $\epsilon$-optimal policy with $\widetilde{O}(\epsilon^{-\frac{d}{\alpha}-2})$ samples in expectation, where $\alpha\in(0,1]$ indicates the smoothness of environment. Compared to previous work, our result exhibits that NPMD can leverage the low-dimensional structure of state space to escape from the curse of dimensionality, explaining the efficacy of deep policy gradient algorithms.