Do Differentiable Simulators Give Better Policy Gradients?

TL;DR

This paper investigates the effectiveness of differentiable simulators for policy gradient estimation in reinforcement learning, highlighting their limitations in complex physical systems and proposing a new hybrid gradient estimator.

Contribution

It introduces an $oldsymbol{oldsymbol{ ext{ extalpha}}}$-order gradient estimator that combines the strengths of first- and zero-order methods, improving robustness and efficiency.

Findings

First-order estimators can be biased or high-variance in complex landscapes.

The $ ext{ extalpha}$-order estimator balances efficiency and robustness.

Numerical examples show the advantages of the $ ext{ extalpha}$-order estimator.

Abstract

Differentiable simulators promise faster computation time for reinforcement learning by replacing zeroth-order gradient estimates of a stochastic objective with an estimate based on first-order gradients. However, it is yet unclear what factors decide the performance of the two estimators on complex landscapes that involve long-horizon planning and control on physical systems, despite the crucial relevance of this question for the utility of differentiable simulators. We show that characteristics of certain physical systems, such as stiffness or discontinuities, may compromise the efficacy of the first-order estimator, and analyze this phenomenon through the lens of bias and variance. We additionally propose an $\alpha$-order gradient estimator, with $\alpha \in [0,1]$, which correctly utilizes exact gradients to combine the efficiency of first-order estimates with the robustness of zero-order methods. We demonstrate the pitfalls of traditional estimators and the advantages of the $\alpha$-order estimator on some numerical examples.