Actor-Critic Reinforcement Learning with Phased Actor

TL;DR

This paper introduces PAAC, a novel actor-critic reinforcement learning method that enhances policy gradient estimation, leading to improved control policies with higher robustness, faster learning, and better performance in continuous control tasks.

Contribution

The paper proposes PAAC, a phased actor-critic approach that improves policy gradient estimation, proves convergence and stability, and demonstrates superior performance over existing methods.

Findings

PAAC reduces variance in policy gradient estimates.

PAAC improves learning speed and robustness.

PAAC outperforms baseline algorithms in control tasks.

Abstract

Policy gradient methods in actor-critic reinforcement learning (RL) have become perhaps the most promising approaches to solving continuous optimal control problems. However, the trial-and-error nature of RL and the inherent randomness associated with solution approximations cause variations in the learned optimal values and policies. This has significantly hindered their successful deployment in real life applications where control responses need to meet dynamic performance criteria deterministically. Here we propose a novel phased actor in actor-critic (PAAC) method, aiming at improving policy gradient estimation and thus the quality of the control policy. Specifically, PAAC accounts for both $Q$ value and TD error in its actor update. We prove qualitative properties of PAAC for learning convergence of the value and policy, solution optimality, and stability of system dynamics. Additionally, we show variance reduction in policy gradient estimation. PAAC performance is systematically and quantitatively evaluated in this study using DeepMind Control Suite (DMC). Results show that PAAC leads to significant performance improvement measured by total cost, learning variance, robustness, learning speed and success rate. As PAAC can be piggybacked onto general policy gradient learning frameworks, we select well-known methods such as direct heuristic dynamic programming (dHDP), deep deterministic policy gradient (DDPG) and their variants to demonstrate the effectiveness of PAAC. Consequently we provide a unified view on these related policy gradient algorithms.