Physical Derivatives: Computing policy gradients by physical forward-propagation

TL;DR

This paper introduces a novel approach to policy gradient computation in reinforcement learning by learning the sensitivity of trajectories to parameter perturbations, enabling model-free policy optimization without explicit dynamic models.

Contribution

It proposes a middle ground method that predicts local system behavior through trajectory sensitivities, reducing reliance on accurate models and demonstrating practical feasibility on a physical robot.

Findings

Method successfully predicts local system behavior

Feasible on a custom-built physical robot

Addresses challenges in applying the approach to real systems

Abstract

Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.