Entropy Augmented Reinforcement Learning

TL;DR

This paper introduces an entropy augmentation technique for reinforcement learning that enhances exploration and performance, especially when combined with on-policy algorithms involving a value critic.

Contribution

It proposes a novel entropy augmentation method that aligns with the soft policy improvement theorem and improves exploration in reinforcement learning.

Findings

Achieves higher rewards on MuJoCo benchmark tasks

Balances exploration and exploitation effectively through temperature control

Enhances exploration bonus in custom environments

Abstract

Deep reinforcement learning was instigated with the presence of trust region methods, being scalable and efficient. However, the pessimism of such algorithms, among which it forces to constrain in a trust region by all means, has been proven to suppress the exploration and harm the performance. Exploratory algorithm such as SAC, while utilizes the entropy to encourage exploration, implicitly optimizing another objective yet. We first observed this inconsistency, and therefore put forward an analogous augmentation technique, which combines well with the on-policy algorithms, when a value critic is involved. Surprisingly, the proposed method consistently satisfies the soft policy improvement theorem, while being more extensible. As the analysis advises, it is crucial to control the temperature coefficient to balance the exploration and exploitation. Empirical tests on MuJoCo benchmark tasks show that the agent is heartened towards higher reward regions, and enjoys a finer performance. Furthermore, we verify the exploration bonus of our method on a set of custom environments.