Guide Actor-Critic for Continuous Control

TL;DR

The paper introduces Guide Actor-Critic (GAC), a novel reinforcement learning method that uses second-order optimization in the action space to improve continuous control policies, outperforming traditional actor-critic approaches.

Contribution

GAC is the first actor-critic method to incorporate second-order optimization in the action space, enhancing policy updates with Hessian-based curvature information.

Findings

GAC outperforms standard actor-critic methods in continuous control tasks.

Deterministic policy gradient is a special case of GAC when Hessians are ignored.

GAC effectively leverages second-order information for improved policy learning.

Abstract

Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC). GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning. Our main theoretical contributions are two folds. First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic. Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored. Through experiments, we show that our method is a promising reinforcement learning method for continuous controls.