Reinforcement learning with experience replay and adaptation of action dispersion

TL;DR

This paper introduces an adaptive method for setting action distribution dispersion in reinforcement learning, balancing exploration and exploitation automatically based on replay buffer actions, improving policy evaluation and convergence.

Contribution

It proposes a novel principle for automatically tuning action dispersion, reducing the need for problem-dependent hyperparameters in reinforcement learning.

Findings

Effective on benchmarks like Ant, HalfCheetah, Hopper, and Walker2D.

Action standard deviations converge to optimized values.

Improves policy evaluation by balancing exploration and exploitation.

Abstract

Effective reinforcement learning requires a proper balance of exploration and exploitation defined by the dispersion of action distribution. However, this balance depends on the task, the current stage of the learning process, and the current environment state. Existing methods that designate the action distribution dispersion require problem-dependent hyperparameters. In this paper, we propose to automatically designate the action distribution dispersion using the following principle: This distribution should have sufficient dispersion to enable the evaluation of future policies. To that end, the dispersion should be tuned to assure a sufficiently high probability (densities) of the actions in the replay buffer and the modes of the distributions that generated them, yet this dispersion should not be higher. This way, a policy can be effectively evaluated based on the actions in the buffer, but exploratory randomness in actions decreases when this policy converges. The above principle is verified here on challenging benchmarks Ant, HalfCheetah, Hopper, and Walker2D, with good results. Our method makes the action standard deviations converge to values similar to those resulting from trial-and-error optimization.