Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning

TL;DR

This paper extends soft policy iteration to include generalized entropy measures, introducing new algorithms TAC and RAC, and develops an ensemble approach EAC that enhances exploration and performance in deep reinforcement learning.

Contribution

The paper presents a new policy iteration theory supporting generalized entropy measures, leading to novel algorithms TAC, RAC, and an ensemble method EAC for improved exploration and learning.

Findings

TAC and RAC outperform SAC on benchmark tasks.

EAC achieves state-of-the-art results with better sample efficiency.

Generalized entropy measures enhance exploration and policy performance.

Abstract

We propose a new policy iteration theory as an important extension of soft policy iteration and Soft Actor-Critic (SAC), one of the most efficient model free algorithms for deep reinforcement learning. Supported by the new theory, arbitrary entropy measures that generalize Shannon entropy, such as Tsallis entropy and Renyi entropy, can be utilized to properly randomize action selection while fulfilling the goal of maximizing expected long-term rewards. Our theory gives birth to two new algorithms, i.e., Tsallis entropy Actor-Critic (TAC) and Renyi entropy Actor-Critic (RAC). Theoretical analysis shows that these algorithms can be more effective than SAC. Moreover, they pave the way for us to develop a new Ensemble Actor-Critic (EAC) algorithm in this paper that features the use of a bootstrap mechanism for deep environment exploration as well as a new value-function based mechanism for high-level action selection. Empirically we show that TAC, RAC and EAC can achieve state-of-the-art performance on a range of benchmark control tasks, outperforming SAC and several cutting-edge learning algorithms in terms of both sample efficiency and effectiveness.