Efficient Gradient Estimation for Motor Control Learning

TL;DR

This paper introduces two novel techniques for reducing gradient estimation errors in noisy reinforcement learning scenarios, specifically applied to motor control tasks, leading to improved learning efficiency and accuracy.

Contribution

It presents two new methods for variance reduction in gradient estimates, extending reinforcement baseline ideas and incorporating variance-based component discounting, applied to motor control learning.

Findings

Significantly improved gradient estimates in noisy motor control tasks

Enhanced learning curves for dart-throwing controller

Effective variance reduction compared to existing methods

Abstract

The task of estimating the gradient of a function in the presence of noise is central to several forms of reinforcement learning, including policy search methods. We present two techniques for reducing gradient estimation errors in the presence of observable input noise applied to the control signal. The first method extends the idea of a reinforcement baseline by fitting a local linear model to the function whose gradient is being estimated; we show how to find the linear model that minimizes the variance of the gradient estimate, and how to estimate the model from data. The second method improves this further by discounting components of the gradient vector that have high variance. These methods are applied to the problem of motor control learning, where actuator noise has a significant influence on behavior. In particular, we apply the techniques to learn locally optimal controllers for a dart-throwing task using a simulated three-link arm; we demonstrate that proposed methods significantly improve the reward function gradient estimate and, consequently, the learning curve, over existing methods.