ADER:Adapting between Exploration and Robustness for Actor-Critic Methods

TL;DR

This paper introduces ADER, a novel reinforcement learning algorithm that dynamically balances exploration and robustness by adjusting value estimation penalties based on uncertainty, improving performance in continuous control tasks.

Contribution

ADER adaptively balances exploration and robustness in actor-critic methods by incorporating a dynamic penalty based on uncertainty, addressing exploration issues and overestimation bias.

Findings

ADER outperforms baseline methods in challenging environments

Dynamic penalty improves exploration without overestimation

Method enhances continuous control task performance

Abstract

Combining off-policy reinforcement learning methods with function approximators such as neural networks has been found to lead to overestimation of the value function and sub-optimal solutions. Improvement such as TD3 has been proposed to address this issue. However, we surprisingly find that its performance lags behind the vanilla actor-critic methods (such as DDPG) in some primitive environments. In this paper, we show that the failure of some cases can be attributed to insufficient exploration. We reveal the culprit of insufficient exploration in TD3, and propose a novel algorithm toward this problem that ADapts between Exploration and Robustness, namely ADER. To enhance the exploration ability while eliminating the overestimation bias, we introduce a dynamic penalty term in value estimation calculated from estimated uncertainty, which takes into account different compositions of the uncertainty in different learning stages. Experiments in several challenging environments demonstrate the supremacy of the proposed method in continuous control tasks.