ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

TL;DR

ELEMENT introduces a multiscale, intrinsically motivated RL framework that enhances exploration without extrinsic rewards, using episodic entropy maximization and efficient kNN-based entropy estimation to improve transfer and scalability.

Contribution

It proposes a novel multiscale entropy optimization, an episodic maximum entropy approach, and a kNN-based entropy estimation method for improved lifelong exploration in RL.

Findings

Outperforms state-of-the-art intrinsic rewards in exploration tasks.

Effective in task-agnostic pre-training and offline RL data collection.

Reduces computational costs significantly compared to previous methods.

Abstract

This paper proposes \emph{Episodic and Lifelong Exploration via Maximum ENTropy} (ELEMENT), a novel, multiscale, intrinsically motivated reinforcement learning (RL) framework that is able to explore environments without using any extrinsic reward and transfer effectively the learned skills to downstream tasks. We advance the state of the art in three ways. First, we propose a multiscale entropy optimization to take care of the fact that previous maximum state entropy, for lifelong exploration with millions of state observations, suffers from vanishing rewards and becomes very expensive computationally across iterations. Therefore, we add an episodic maximum entropy over each episode to speedup the search further. Second, we propose a novel intrinsic reward for episodic entropy maximization named \emph{average episodic state entropy} which provides the optimal solution for a theoretical upper bound of the episodic state entropy objective. Third, to speed the lifelong entropy maximization, we propose a $k$ nearest neighbors ($k$NN) graph to organize the estimation of the entropy and updating processes that reduces the computation substantially. Our ELEMENT significantly outperforms state-of-the-art intrinsic rewards in both episodic and lifelong setups. Moreover, it can be exploited in task-agnostic pre-training, collecting data for offline reinforcement learning, etc.