Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies

TL;DR

This paper introduces GProp, a novel deep reinforcement learning algorithm that effectively learns continuous policies by estimating value function gradients and employing a three-network DAC model, excelling in complex tasks.

Contribution

The paper presents GProp, combining a new TD-based gradient learning method with the deviator-actor-critic model for improved continuous policy learning in reinforcement learning.

Findings

GProp performs competitively on a nonparametric regression-based bandit task.

GProp achieves state-of-the-art results on the octopus arm benchmark.

The method accurately estimates value function gradients in complex environments.

Abstract

This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-critic (DAC) model, which comprises three neural networks that estimate the value function, its gradient, and determine the actor's policy respectively. We evaluate GProp on two challenging tasks: a contextual bandit problem constructed from nonparametric regression datasets that is designed to probe the ability of reinforcement learning algorithms to accurately estimate gradients; and the octopus arm, a challenging reinforcement learning benchmark. GProp is competitive with fully supervised methods on the bandit task and achieves the best performance to date on the octopus arm.