# Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems

## TL;DR

This paper develops a theoretical framework showing that for nearly deterministic systems, finite-difference policy gradient estimates can have lower variance than traditional methods, with empirical validation on an inverted pendulum.

## Contribution

It introduces a new theoretical understanding of policy gradient estimation in nearly deterministic systems, highlighting the advantages of finite-difference methods.

## Key findings

- Finite-difference estimates have lower variance in nearly deterministic systems.
- Theoretical analysis explains the effectiveness of finite-difference methods.
- Empirical results on the inverted pendulum support the theory.

## Abstract

Reinforcement learning is a promising approach to learning robotics controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomenon. Our key insight is that many dynamical systems (especially those of interest in robotics control tasks) are nearly deterministic -- i.e., they can be modeled as a deterministic system with a small stochastic perturbation. We show that for such systems, finite-difference estimates of the policy gradient can have substantially lower variance than estimates based on the policy gradient theorem. Finally, we empirically evaluate our insights in an experiment on the inverted pendulum.