Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action

TL;DR

This paper introduces the Instant Retrospect Action (IRA) algorithm, which enhances policy exploitation in online reinforcement learning by improving representation learning, policy constraints, and update frequency, leading to better efficiency and performance.

Contribution

The paper proposes IRA, combining RDE, GAG, and IPU mechanisms to accelerate policy updates and improve learning in online RL, addressing exploration and overestimation issues.

Findings

IRA significantly improves learning efficiency on MuJoCo tasks.

IRA achieves higher final performance compared to baseline algorithms.

Early-stage conservatism reduces overestimation bias.

Abstract

Existing value-based online reinforcement learning (RL) algorithms suffer from slow policy exploitation due to ineffective exploration and delayed policy updates. To address these challenges, we propose an algorithm called Instant Retrospect Action (IRA). Specifically, we propose Q-Representation Discrepancy Evolution (RDE) to facilitate Q-network representation learning, enabling discriminative representations for neighboring state-action pairs. In addition, we adopt an explicit method to policy constraints by enabling Greedy Action Guidance (GAG). This is achieved through backtracking historical actions, which effectively enhances the policy update process. Our proposed method relies on providing the learning algorithm with accurate $k$-nearest-neighbor action value estimates and learning to design a fast-adaptable policy through policy constraints. We further propose the Instant Policy Update (IPU) mechanism, which enhances policy exploitation by systematically increasing the frequency of policy updates. We further discover that the early-stage training conservatism of the IRA method can alleviate the overestimation bias problem in value-based RL. Experimental results show that IRA can significantly improve the learning efficiency and final performance of online RL algorithms on eight MuJoCo continuous control tasks.The code is available at https://github.com/2706853499/IRA.