How are policy gradient methods affected by the limits of control?

TL;DR

This paper examines how control-theoretic limitations impact stochastic policy gradient methods, revealing issues like noisy estimates and the curse of dimensionality in certain system classes.

Contribution

It provides a control-theoretic analysis of policy gradient methods, highlighting conditions causing noise and complexity issues not previously well-understood.

Findings

Ill-conditioned systems lead to noisy gradient estimates.

Stable systems can still suffer from the curse of dimensionality.

Results apply to both fully observed and partially observed systems.

Abstract

We study stochastic policy gradient methods from the perspective of control-theoretic limitations. Our main result is that ill-conditioned linear systems in the sense of Doyle inevitably lead to noisy gradient estimates. We also give an example of a class of stable systems in which policy gradient methods suffer from the curse of dimensionality. Our results apply to both state feedback and partially observed systems.