Addressing Function Approximation Error in Actor-Critic Methods

TL;DR

This paper identifies and addresses function approximation errors in actor-critic reinforcement learning, proposing novel techniques to reduce overestimation bias and improve policy performance across various tasks.

Contribution

It introduces mechanisms based on Double Q-learning and delayed policy updates to mitigate overestimation bias in actor-critic methods.

Findings

Outperforms state-of-the-art methods on OpenAI gym tasks

Reduces overestimation bias in actor-critic algorithms

Improves policy quality and learning stability

Abstract

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.